Systems and methods for headphone equalization and room adaptation for binaural reproduction in augmented reality

Abstract

A system is provided that includes an analyzer (152) for determining a plurality of binaural room impulse responses and a loudspeaker signal generator (154) for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source, wherein the analyzer (152) is configured to determine the plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account effects resulting from a user wearing headphones.

Description

[Technical field]

[0001] The present invention relates to headphone equalization and room adaptation for binaural reproduction in augmented reality (AR). [Background technology]

[0002] Selective hearing (SH) refers to the ability of a listener to direct attention to a particular sound source or sources in an auditory scene, which in turn means that the listener's focus on sources of no interest is reduced.

[0003] Thus, human listeners can communicate in large environments, usually by exploiting different aspects, namely, time and level differences that are direction-dependent when listening with two ears, as well as different spectral colorings of sound that are direction-dependent. The latter allows the auditory sense to determine the direction of sound sources and separate different sound sources, even when listening with one ear.

[0004] The exact location of the sound source cannot be determined by time and level differences alone. decision It is not enough to measure the time difference and the level difference, and the positions with the same time and level difference are located on the hyperbolic surface. decision The ambiguity is called the cone of confusion. In a room, each sound source is reflected by boundary surfaces. Each of these so-called mirror sources is located on a further hyperbolic surface. Human hearing combines information about the direct sound and the associated reflections into an auditory event, thus resolving the confusion ambiguity. At the same time, the reflections belonging to a sound source increase the perceived loudness of the sound source.

[0005] Also, in the case of natural sound sources, especially speech, signal parts of different frequencies are combined in time. In binaural hearing, all of these aspects are used together. Moreover, large disturbance sources that can be well localized can be actively ignored, so to speak.

[0006] In the literature, the concept of selective hearing is related to other terms such as assisted hearing [1], virtual hearing environments and amplified hearing environments [2]. Assisted hearing is a broader term that includes virtual, amplified and SH applications.

[0007] According to the prior art, classical hearing devices operate mainly in a monophonic manner, i.e. the signal processing for the left and right ears is completely independent in terms of frequency response and dynamic compression, as a result of which the time, level and frequency differences between the ear signals are lost.

[0008] Modern so-called binaural hearing devices combine the correction factors of two hearing devices. They often have several microphones, but usually only the microphone with the "most speech-like" signal is selected and no explicit beamforming is calculated. In complex hearing situations, the desired and undesired sound signals are amplified in the same way, and therefore focusing on the desired sound components is not supported.

[0009] In the field of hands-free devices, for example for telephone, several microphones are already used today, so-called beams are calculated from the individual microphone signals, sounds coming from the direction of the beam are amplified and sounds from other directions are reduced. Today's methods learn constant sounds in the background (e.g. engine and wind noise in a car) and large disturbances that are well localizable via further beams and subtract these from the used signal (e.g. generalized sidelobe canceller). Sometimes telephone systems use detectors that detect static characteristics of the voice and suppress everything that is not structured like voice. In hands-free devices, only mono signals are finally transmitted, losing interesting spatial information in the transmission path to capture the situation and to provide the illusion as if "one was there", especially in the case of several talkers having mutual calls. By suppressing non-voice signals, important information about the acoustic environment of the other party is lost, which can hinder communication.

[0010] Inherently, humans are capable of consciously focusing on "selective listening" to individual sound sources in the environment. An automated system for selective hearing using artificial intelligence (AI) must first learn the underlying concepts. Automatic decomposition of an acoustic scene (scene decomposition) first requires the detection and classification of all active sound sources, and then separating them so that they can be further processed, amplified, or attenuated as separate audio objects.

[0011] The research field of auditory scene analysis seeks to detect and classify, based on recorded audio signals, temporally located sound events such as steps, clapping, or voices, as well as more global acoustic scenes such as concerts, restaurants, or supermarkets. In this case, current methods exclusively use methods from the fields of artificial intelligence (AI) and deep learning. This involves data-driven learning of deep neural networks that learn on a large training volume to detect characteristic patterns within audio signals

[70] . Inspired by progress in the research fields of image processing (computer vision) and speech processing (natural language processing), among others, a mixture of convolutional neural networks for two-dimensional pattern detection in spectrogram representations and recurrent layers (recurrent neural networks) for temporal modeling of sounds is used in principle.

[0012] Speech analysis has a particular set of challenges to address. Due to their complexity, deep learning models are very data intensive. In contrast to the research fields of image processing and speech processing, the datasets available for speech processing are relatively small. The largest dataset is the AudioSet dataset from Google

[83] with about 2 million sound examples and 632 different sound event classes, and most datasets used in research are significantly smaller. This small amount of training data can be addressed, for example, using transfer learning, where models pre-trained on large datasets can then be adapted to the use case. decisionThe new classes are then fine-tuned on a smaller dataset (fine-tuning)

[77] . Furthermore, methods from semi-supervised learning are often used to include unannotated speech data, which is commonly available in large quantities, in training.

[0013] A further important difference compared to image processing is that in the case of simultaneously heard sound events, there is no masking of sound objects (as in the case of images) but rather a complex phase-dependent overlap. Current algorithms in deep learning use so-called "attention" mechanisms, which allow, for example, the model to focus classification on specific time segments or frequency ranges

[23] . The detection of sound events is further complicated by the high variance regarding their duration. Algorithms should be able to robustly detect very short events, such as gunshots, and long events, such as a passing train.

[0014] Due to the strong dependence of the models on the acoustic conditions in the recording of the training data, the models often show unexpected behavior in new acoustic environments, which are different, for example, in terms of spatial reverberation or microphone positioning. Various solution techniques have been developed to mitigate this problem. For example, data augmentation methods try to achieve higher robustness and invariance of the models through the simulation of different acoustic conditions

[68] and the artificial overlap of different sound sources. Furthermore, the parameters of complex neural networks can be adjusted in different ways so that overtraining and specialization on the training data is avoided and at the same time a better generalization to unseen data is achieved. In recent years, different algorithms have been proposed for "domain adaptation"

[67] to adapt previously trained models to new application conditions. For the in-headphone usage scenarios planned in this project, the real-time capability of the sound source detection algorithm is of fundamental importance. Here, a trade-off must necessarily be made between the complexity of the neural network and the maximum possible number of computational operations on the underlying computing platform. Even if the duration of a sound event is long, it needs to be detected as quickly as possible in order to start the corresponding sound source separation.

[0015] At the Fraunhofer IDMT, a lot of research has been carried out in recent years in the field of automatic sound source detection. In the research project "Stadtram", a distributed sensor network was developed that can measure noise levels and classify between 14 different acoustic scenes and event classes based on audio signals recorded at different locations in a city

[69] . In this case, the processing at the sensors is performed in real time on the embedded platform Raspberry Pi 3. In previous work, a novel method for data compression of spectrograms based on autoencoder networks was considered

[71] . Recently, through the use of methods from deep learning in the field of music signal processing (music information retrieval), great progress has been made in applications such as music transcription

[76] ,

[77] , chord detection

[78] , and instrument detection

[79] . In the field of industrial sound processing, new datasets have been established and deep learning methods are being used, for example, to monitor the acoustic condition of electric motors

[75] .

[0016] The scenario addressed in this embodiment assumes several sound sources whose number and type are initially unknown and may change continuously. For source separation, several sound sources with similar characteristics, such as several loudspeakers, are a particularly big challenge

[80] .

[0017] To achieve high spatial resolution, several microphones must be used in the form of an array

[72] . In contrast to conventional sound recordings, which are mono (one channel) or stereo (two channels), such a recording scenario allows for precise localization of sound sources around the listener.

[0018] Source separation algorithms typically leave behind artifacts such as distortion and crosstalk between the sources [5], which can generally be perceived as disruptive by the listener. By remixing the tracks, such artifacts can be partially masked and therefore reduced

[10] .

[0019] To enhance "blind" source separation, additional information such as the detected number and type of sound sources or their estimated spatial location is often used (informed source separation

[74] ). For conferences with several active speakers, current analysis systems estimate the number of speakers simultaneously and their respective temporal activity. decision They can then be separated by source separation

[66] .

[0020] At the Fraunhofer IDMT, much work has been done in recent years on perception-based evaluation of sound source separation algorithms

[73] .

[0021] In the field of music signal processing, real-time capable algorithms have been developed for separating solo and accompanying instruments, utilizing the base frequency estimate of the solo instrument as additional information

[81] . An alternative approach for separating singing from complex musical pieces based on deep learning methods is proposed in

[82] . Specialized source separation algorithms have also been developed for applications in the context of industrial audio analysis [7].

[0022] Headphones greatly affect the acoustic perception of the surroundings. Depending on the headphone's construction, the sound incident towards the ear is attenuated to different degrees. In-ear headphones completely block the ear channel

[85] . Closed headphones that surround the pinna also strongly isolate the listener acoustically from the external environment. Open and semi-open headphones allow sound to pass through completely or partially

[84] . In many applications in daily life, it is desirable for headphones to isolate unwanted surrounding sounds more strongly than is possible for their construction type.

[0023] Furthermore, the effects of external interference can be attenuated by active noise control (ANC). This is achieved by recording the incoming sound signals by the microphones of the headphones and then reproducing them by the loudspeakers so that these sound parts and the sound parts penetrating the headphones cancel each other by interference. Overall, this can achieve a strong acoustic isolation from the surroundings. However, in many everyday situations this is associated with dangers, so it is desirable to be able to intelligently turn on this function on demand.

[0024] The first product allows the microphone signal to be passed through the headphones to reduce passive isolation. So, besides the prototype

[86] , there are already commercial products advertising the feature of "transparent hearing". For example, Sennheiser offers the functionality with its AMBEO headset

[88] , and Bragi with its product "Dash Pro". However, this possibility is only the beginning. In the future, this functionality should be significantly expanded to be able to make individual signal parts (for example only the voice signal or the alarm signal) exclusively hearable on demand, in addition to turning the ambient sound completely on and off. The French company Orosound allows the person wearing the headset "chilled earphones" to adapt the strength of the ANC by means of a slider

[89] . In addition, the voice of the conversation partner can also be induced during the activated ANC. However, this only works if the conversation partner is located face to face within a 60° cone. Direction-independent adaptation is not possible.

[0025] US2015195641 (see

[91] ) discloses a method implemented to generate an auditory environment for a user, the method comprising receiving a signal representative of a user's ambient auditory environment and processing the signal using a microprocessor to identify at least one sound type of a plurality of sound types in the ambient auditory environment. The method further comprises receiving a user preference for each of the plurality of sound types, modifying the signal for each sound type in the ambient auditory environment, and outputting the modified signal to at least one loudspeaker to generate the auditory environment for the user.

[0026] Headphone equalization and room adaptation (or spatial / spatial adaptation or spatial / spatial compensation) for binaural reproduction in augmented reality (AR) are important issues.

[0027] In a typical scenario, a human listener wears acoustically (partially) transparent headphones through which he hears ambient sounds. Furthermore, an additional sound source is played through the headphones, said sound source being embedded in the real surroundings in such a way that the listener cannot distinguish between the real acoustic scene and the additional sound.

[0028] Typically, the direction in which the head rotates and the position of the listener in the room (or space) are tracked (6 degrees of freedom (6DoF) decision It is known from research that better results (i.e. externalization and accurate localization) are achieved when the room acoustics of the recording and playback rooms match, or when the recording is adapted to the playback room.

[0029] In this case, an exemplary solution can be implemented as follows.

[0030] In a first step, measurements of the BRIR are performed in an individualized manner without headphones or using an artificial head with a probe microphone.

[0031] In the second step, an analysis of the room characteristics of the recording chamber is performed based on the measured BRIR.

[0032] In a third step, measurements of the headphone transfer functions are made using an artificial head, either in an individualized way or by means of a co-located probe microphone. This allows the equalization functions to be calculated. decision will be done.

[0033] Optionally, in a fourth step, measuring the room characteristics of the reproduction room, analysing the acoustic characteristics of the reproduction room and adapting the BRIR to the reproduction room may be performed.

[0034] Then, in a further step, a convolution (or folding) of the source, augmented with a correctly positioned and optionally adapted BRIR, is performed to obtain two raw channels: a convolution of the raw channels with an equalization function to obtain a headphone signal;

[0035] Finally, in a further step, playback of the headphone signal is performed via the headphones.

[0036] However, the problem is that when headphones are worn, the effect of the pinna on the BRIR is eliminated, i.e. the BRIR is different than without the headphones. This results in a different natural sound source being heard than without the headphones, but a virtual augmented sound source being reproduced as if the headphones were not there. [Prior art documents] [Patent documents]

[0037] [Patent Document 1] US Patent Publication No. 2015195641 Summary of the Invention [Problem to be solved by the invention]

[0038] Simple, fast and efficient room characterization of the regeneration room decision It is desirable to provide a concept that enables [Means for solving the problem]

[0039] Embodiments of the present invention are provided below.

[0040] Thus, claim 1 provides a system according to an embodiment of the invention, claim 19 provides a method and claim 20 provides a computer program.

[0041] A system according to an embodiment of the present invention generates multiple binaural room impulse responses. decision and a loudspeaker signal generator for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to a source signal of the at least one sound source. The analyzer analyzes the plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account an effect resulting from the headphones being worn by a user. decision The device is configured to:

[0042] Further provided is a method according to an embodiment of the present invention, the method comprising: Multiple binaural room impulse responses decision and - generating at least two loudspeaker signals in response to a plurality of binaural room impulse responses and in response to a source signal of at least one sound source; Includes.

[0043] The plurality of binaural room impulse responses are arranged such that each of the plurality of binaural room impulse responses takes into account the effect caused by a user wearing headphones. decision will be done.

[0044] Furthermore, there is provided a computer program according to an embodiment of the invention having a program code for performing the method described above.

[0045] Next, preferred embodiments of the present invention will be described with reference to the drawings. [Brief description of the drawings]

[0046] [Figure 1] FIG. 1 illustrates a system according to one embodiment. [Diagram 2] FIG. 1 illustrates a further system for assisting selective hearing according to a further embodiment. [Diagram 3] FIG. 1 illustrates a further system for assisting selective hearing, further including a user interface. [Figure 4] FIG. 1 illustrates a system for assisting selective hearing that includes a hearing device having two corresponding loudspeakers. [Figure 5a] FIG. 1 illustrates a system for assisting selective hearing, including a housing structure and two loudspeakers. [Figure 5b] FIG. 1 illustrates a system for assisting selective hearing that includes headphones with two loudspeakers. [Figure 6] FIG. 1 illustrates a system according to one embodiment including a remote device 190 including a detector and locator as well as an audio type classifier and a signal portion modifier and signal generator. [Figure 7] FIG. 1 illustrates a system according to one embodiment, including five subsystems. [Figure 8] FIG. 2 illustrates a corresponding scenario according to one embodiment. [Figure 9] FIG. 1 illustrates a scenario according to an embodiment with four external sound sources. [Figure 10] FIG. 1 is a diagram showing a processing workflow for SH applications according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0047] FIG. 1 is a diagram illustrating a system according to one embodiment.

[0048] The system generates multiple binaural room impulse responses. decision The analyzer 152 includes an analyzer 152 for analyzing the

[0049] Furthermore, the system includes a loudspeaker signal generator 154 for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to the source signal of the at least one sound source.

[0050] The analyzer 152 analyzes the plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account the effect caused by the user wearing headphones. decision The device is configured to:

[0051] In one embodiment, for example, the system may include headphones, which may be configured to output at least two loudspeaker signals.

[0052] According to one embodiment, for example, a headphone may include at least two headphone capsules and at least one microphone for measuring sound in each of the two headphone capsules, for example, the at least one microphone for measuring sound may be disposed in each of the two headphone capsules, where, for example, the analyzer 152 may use the measurements of the at least one microphone in each of the two headphone capsules to generate a plurality of binaural room impulse responses. decision Headphones intended for binaural reproduction always comprise at least two headphone capsules (e.g. for different frequency ranges), and three or more capsules may be provided.

[0053] In one embodiment, for example, at least one microphone in each of the two headphone capsules generates one or more recordings of the sound situation in the reproduction room (or space) prior to reproduction of the at least two loudspeaker signals by the headphones, and estimates the raw audio signal of at least one sound source from the one or more recordings. decision Then, the binaural room impulse responses of multiple binaural room impulse responses for the sound sources in the playback room are decision The device may be configured to:

[0054] According to one embodiment, for example, at least one microphone of each of the two headphone capsules generates one or more further recordings of the sound situation in the reproduction room during playback of the at least two loudspeaker signals by the headphones, subtracts the augmented signal from these one or more further recordings, and obtains an estimate of the raw sound signal from the one or more sound sources. decision Then, the binaural room impulse responses of multiple binaural room impulse responses to the sound sources in the playback room are decision The device may be configured to:

[0055] In one embodiment, for example, the analyzer 152 may analyze the acoustic room characteristics of the reproduction room. decision and may be configured to adapt multiple binaural room impulse responses depending on the acoustic room characteristics.

[0056] According to one embodiment, for example, at least one microphone may be placed in each of the two headphone capsules to measure sounds near the entrance of the ear canal.

[0057] In one embodiment, for example, the system may include one or more additional microphones outside the two headphone capsules to measure the sound conditions in the playback room.

[0058] According to one embodiment, for example, the headphones are Headphone band For example, at least one of the one or more further microphones may include Headphone band is placed on top.

[0059] In one embodiment, for example, the loudspeaker signal generator 154 may be configured to generate at least two loudspeaker signals by convolving each of a plurality of binaural room impulse responses with a source signal of a plurality of one or more source signals.

[0060] According to one embodiment, for example, the analyzer 152 may generate at least one of the multiple binaural room impulse responses (or some or all of the binaural room impulse responses) in response to the movement of the headphones. decision The device may be configured to:

[0061] In an embodiment, the system detects the movement of the headphones. decision For example, the sensor may be an acceleration pickup or other sensor with at least 3DoF (three degrees of freedom) to capture head rotation. For example, a 6DoF sensor (six degrees of freedom sensor) may be used.

[0062] Certain embodiments of the present invention address the technical challenge that certain sounds in the auditory environment are often very loud, are distracting, and selective hearing is desired. While the human brain itself can perform selective hearing to some extent, intelligent technological assistants can significantly improve selective hearing. Just as glasses help many people in modern life to better perceive their environment, there are hearing aids for hearing, but even people with normal hearing can benefit from assistance from intelligent systems in many situations. To realize an "intelligent hearable" (hearing device or hearing aid), a technical system needs to analyze the (acoustic) environment and identify individual sound sources so that they can be processed separately. Research into this problem has already been conducted, but the prior art has not achieved the analysis and processing of the entire acoustic environment in real time (transparent to the ear) and with high sound quality (content that sounds indistinguishable from the normal acoustic environment).

[0063] An improved concept for machine listening is provided below.

[0064] In the first step, measurements of the BRIR using headphones are performed either individually using a probe microphone or using headphones.

[0065] In the second step, an analysis of the room characteristics of the recording chamber is performed based on the measured BRIR.

[0066] Optionally, for example in a third step, before playback, at least one built-in microphone in each shell records the actual sound situation in the playback room. From these recordings, estimates of the raw sound signals of one or more sound sources are obtained. decision The BRIR of each sound source in the playback room / sound source is decision From this estimation, the acoustic characteristics of the playback room are decision The BRIR in the recording room is then adapted to it.

[0067] Optionally, for example in a further step, during playback at least one built-in microphone in each shell records the actual sound situation in the playback room. From these recordings, the augmented signal is first subtracted and then an estimate of the raw sound signal of one or more sound sources is obtained. decision The BRIR of each sound source in the playback room / sound source is decision From this estimation, the acoustic characteristics of the playback room are decision and the BRIR of the playback room is adapted to it.

[0068] In a further step, a convolution of the source augmented with the correctly positioned and optionally adapted BRIR is performed to obtain the headphone signal.

[0069] Finally, in a further step, playback of the headphone signal is performed via the headphones.

[0070] In one embodiment, for example, at least one microphone is placed within each headphone capsule to measure sounds near the entrance of the ear canal.

[0071] According to one embodiment, additional microphones are optionally installed outside the headphones, possibly to measure and analyze the sound situation in the reproduction room. Headphone band It is also placed on the top of the

[0072] In an embodiment, identical natural and extended source sounds are realized.

[0073] The embodiment recognises that measuring the characteristics of the headphones is not necessary.

[0074] Thus, the embodiments provide a concept for measuring the room characteristics of a reproduction room.

[0075] Some embodiments provide starting values ​​and (post)optimization of room adaptation. The provided concept also works when the room acoustics of the reproduction room change, for example when the listener moves to another room (or space).

[0076] Among other things, the embodiments are based on installing different technologies for assisting hearing in a technical system and then combining them in such a way that an improvement in sound and quality of life (e.g. desired sounds are louder, undesired sounds are softer, speech intelligibility is better) is achieved for people with normal hearing and for people with hearing loss.

[0077] FIG. 2 is a diagram illustrating a system for assisting selective hearing according to one embodiment.

[0078] The system includes a detector 110 for detecting source signal portions of one or more sound sources by using at least two received microphone signals of the auditory environment (or listening environment).

[0079] Additionally, the system includes a location calculator for assigning location information to each of the one or more sound sources. decision The device includes a container 120.

[0080] Additionally, the system includes an audio type classifier 130 for assigning an audio signal type to the audio source signal portions of each of the one or more audio sources.

[0081] Further, the system includes a signal portion modifier 140 for modifying the source signal portion of at least one of the one or more sound sources depending on the sound signal type of the source signal portion of the at least one sound source to obtain a modified sound signal portion of the at least one sound source.

[0082] The analyzer 152 and the loudspeaker signal generator 154 of FIG.

[0083] The analyzer 152 of the signal generator 150 is configured to generate a plurality of binaural room impulse responses for each of the one or more sound sources that depend on the position information of this sound source and the orientation of the user's head.

[0084] The loudspeaker signal generator 154 of the signal generator 150 is configured to generate at least two loudspeaker signals in response to a plurality of binaural room impulse responses and in response to the modified audio signal portion of the at least one sound source.

[0085] According to one embodiment, for example, the detector 110 may be configured to detect source signal portions of one or more sound sources by using a deep learning model.

[0086] In one embodiment, for example, the position decision The device 120 may generate position information for each of the one or more sound sources in response to the captured image or recorded video. decision The device may be configured to:

[0087] According to one embodiment, for example, the position decision The device 120 detects lip movements of a person in the video, and assigns position information to a sound source signal portion of one of the one or more sound sources according to the lip movements, thereby obtaining position information according to the video for each of the one or more sound sources. decision The device may be configured to:

[0088] In one embodiment, for example, the detector 110 determines one or more acoustic characteristics of the auditory environment in response to at least two received microphone signals. decision The device may be configured to:

[0089] According to one embodiment, for example, the signal generator 150 generates a plurality of binaural room impulse responses in response to one or more acoustic characteristics of the hearing environment. decision The device may be configured to:

[0090] In one embodiment, for example, the signal portion modifier 140 may be configured to select at least one sound source whose sound source signal portion is to be modified in response to a previously learned user scenario and to modify it in response to the previously learned user scenario.

[0091] According to one embodiment, for example, the system may include a user interface 160 for selecting a previously learned user scenario from a group of two or more previously learned user scenarios. FIG. 3 illustrates such a system according to one embodiment, further including such a user interface 160.

[0092] In one embodiment, for example, the detector 110 and / or the position decision The decoder 120 and / or the voice type classifier 130 and / or the signal modifier 140 and / or the signal generator 150 may be configured to perform parallel signal processing using a Hough transform, or using multiple VLSI chips, or by using multiple memristors.

[0093] According to one embodiment, for example, the system may include a hearing device 170 that functions as a hearing aid for a user with limited hearing capabilities and / or impaired hearing, the hearing device including at least two loudspeakers 171, 172 for outputting at least two loudspeaker signals. Figure 4 shows such a system according to one embodiment, including such a hearing device 170 with two corresponding loudspeakers 171, 172.

[0094] In one embodiment, for example, the system may include at least two loudspeakers 181, 182 for outputting at least two loudspeaker signals and a housing structure 183 for accommodating the at least two loudspeakers, the at least one housing structure 183 being suitable for being fixed to a user's head 185 or any other body part of the user. Figure 5a shows such a housing structure 183 and a corresponding system including two loudspeakers 181, 182.

[0095] According to one embodiment, for example, the system may include a headphone 180 including at least two loudspeakers 181, 182 for outputting at least two loudspeaker signals. Figure 5b shows a corresponding headphone 180 having two loudspeakers 181, 182 according to one embodiment.

[0096] In one embodiment, for example, the detector 110 and the position decision The decoder 120 as well as the sound type classifier 130 as well as the signal portion modifier 140 and the signal generator 150 may be integrated into a headphone 180.

[0097] According to one embodiment shown in FIG. 6, for example, the system includes a detector 110 and a position decision The remote device 190 may include the speech recognition device 120 and the audio type classifier 130 as well as the remote device 190 including the signal portion modifier 140 and the signal generator 150. In this case, for example, the remote device 190 may be spatially separated from the headphones 180.

[0098] In one embodiment, for example, the remote device 190 may be a smartphone.

[0099] The embodiments do not necessarily use microprocessors, but rather use parallel signal processing steps such as Hough transforms, VLSI chips or memristors for energy-efficient implementations, especially also for artificial neural networks.

[0100] In an embodiment, the auditory environment is captured and reproduced spatially, using two or more signals for the representation of the input signal on the one hand, and also using spatial reproduction on the other hand.

[0101] In an embodiment, the signal separation is performed by a deep learning (DL) model (e.g. CNN, RCNN, LSTM, Siamese network) that processes information from at least two microphone channels simultaneously, with at least one microphone in each hearable. According to the invention, several output signals (corresponding to individual sound sources) are separated by cross analysis together with their respective spatial positions. decision If a recording means (microphone) is connected to the head, the position of the object changes with the movement of the head. This makes it possible to naturally focus on important / unimportant sounds, for example by facing the sound object.

[0102] In some embodiments, the algorithm for signal analysis is based on, for example, deep learning architectures. Alternatively, it uses variation by analysis unit or variation by separate networks for aspect localization, detection and sound separation. An alternative use of generalized cross-correlation (correlation vs. time offset) accommodates frequency-dependent shadowing / separation by the head and improves localization, detection and sound source separation.

[0103] According to one embodiment, different source categories (e.g., speech, vehicle, male / female / voice of child, warning tone, etc.) are learned by the detector in a training phase, where the source separation network is also trained on the target stimuli for high signal quality as well as the localization network with high accuracy of localization.

[0104] For example, the training steps described above use multi-channel speech data, and a first training round is typically performed in a laboratory using simulated or recorded speech data, followed by training runs in different natural environments (e.g., living rooms, classrooms, train stations, (industrial) production environments, etc.), i.e., transfer learning and domain adaptation are performed.

[0105] Alternatively or additionally, the position detector may also detect the visual position of the sound source / sound source. decision In the case of speech, lip movements are correlated with the audio signal coming from the sound source separator to achieve more accurate localization.

[0106] After training, a DL model exists that has the network architecture and associated parameters.

[0107] In some embodiments, the auralization is performed by binaural synthesis, which offers the additional advantage that, although it is not possible to completely remove undesirable components, it is possible to reduce them to a level that is perceptible but not disturbing. This has the additional advantage of perceiving unexpected additional sources (warning signals, speech bubbles, ...) that would be missed if they were completely turned off.

[0108] According to some embodiments, the analysis of the auditory environment is used not only to separate objects, but also to analyze acoustic characteristics (e.g., reverberation time, initial time gap). These characteristics are then used in binaural synthesis to adapt pre-stored (possibly individualized) binaural room impulse responses (BRIR) to the real room (or space). By reducing the room divergence, the listener has a significantly reduced listening effort when understanding the optimized signal. Minimizing the room spread affects the externalization of auditory events and therefore the authenticity of spatial audio reproduction in the monitoring room. For speech understanding or for general understanding of the optimized signal, there is no known solution in the prior art.

[0109] In an embodiment, a user interface is used to determine which sound source is selected. decision According to the present invention, this is done by pre-learning different user scenarios such as "amplify voice directly in front" (conversation with one person), "amplify voice in ±60 degree range" (group conversation), "suppress music and amplify music" (don't want to hear concert-goers), "silence everything" (want to be left alone), "suppress all voices and warning tones", etc.

[0110] Some embodiments are independent of the hardware used, i.e., open and closed headphones can be used. The signal processing can be built into the headphones, an external device, or the smartphone. Optionally, in addition to playing acoustically recorded and processed signals, signals can be played directly from the smartphone (e.g., music, phone calls).

[0111] In another embodiment, an ecosystem for "AI-assisted selective listening" is provided. The embodiment refers to "personalized auditory reality" (PARty). In such a personalized environment, the listener can amplify, reduce or modify defined acoustic objects. A series of analysis and synthesis processes should be performed to create a wholesome experience adapted to individual requirements. Research into targeted transformation stages forms an essential component for this.

[0112] Some embodiments provide analysis of the real sound environment and detection of individual acoustic objects, isolation, tracking and editability of the available objects, as well as reconstruction and playback of modified acoustic scenes.

[0113] In an embodiment, detection of sound events, isolation of sound events, and suppression of some sound events are provided.

[0114] In an embodiment, AI methods (particularly deep learning based methods) are used.

[0115] Embodiments of the present invention contribute to the development of technologies for spatial audio recording, signal processing, and playback.

[0116] For example, embodiments create spatiality and three-dimensionality in a multimedia system with interacting users.

[0117] In this case, the embodiment is based on research knowledge of the perceptual and cognitive processing of spatial hearing / listening.

[0118] Some embodiments use two or more of the following concepts.

[0119] Scene Decomposition: This involves spatial acoustic detection of the real environment, as well as parameter estimation and / or position-dependent sound field analysis.

[0120] Scene Representation: This involves the representation and identification, and / or efficient representation and storage of objects and / or environments.

[0121] Scene combination and reproduction: This includes the adaptation and transformation of objects and environments, and / or the rendering and auralization.

[0122] Quality assessment: This includes technical and / or auditory quality measurements.

[0123] Microphone positioning: This involves the application of a microphone array and appropriate audio signal processing.

[0124] Signal conditioning: This includes feature extraction as well as dataset generation for machine learning (ML).

[0125] Estimation of room and ambient acoustics: This involves in-situ measurement and estimation of room acoustic parameters and / or providing room acoustic signatures for source separation and ML.

[0126] Auralization: This includes spatial audio reproduction with auditory adaptation to the environment and / or validation and evaluation and / or functional proof and quality estimation.

[0127] FIG. 8 illustrates a corresponding scenario according to one embodiment.

[0128] The embodiments combine concepts for sound source detection, classification, separation, localization, and enhancement, highlighting recent advances in each field and illustrating the connections between them.

[0129] In the following, we provide a consistent concept that can combine / detect / classify / localize and separate / enhance sound sources to provide the flexibility and robustness required for real-world SH.

[0130] Furthermore, the embodiments provide a low latency concept suitable for real-time performance when dealing with the dynamics of real auditory scenes.

[0131] Some embodiments use concepts from deep learning, machine hearing, and smart headphones (smart hearables) to enable the listener to selectively modify the auditory scene.

[0132] The embodiments offer the listener the possibility to selectively enhance, attenuate, suppress or modify sound sources within an auditory scene using hearing devices such as headphones, earphones etc.

[0133] FIG. 9 illustrates a scenario according to an embodiment with four external sound sources.

[0134] In Fig. 9, the user is the center of the auditory scene. In this case, four external sound sources (S1 to S4) are active around the user. The user interface allows the listener to influence the auditory scene. Sources S1 to S4 can be attenuated, improved or suppressed by their corresponding sliders. As can be seen in Fig. 2, the listener can define sound sources or sound events that should be kept in the auditory scene or suppressed from it. In Fig. 2, the urban background noise should be suppressed, but alarms or phone calls should be kept. At all times, the user has the possibility to play (or play) additional audio streams such as music or radio through the auditory device.

[0135] The user is usually the center of the system and controls the auditory scene by means of a control unit. The user can modify the auditory scene by means of a user interface as shown in Fig. 9 or by means of any type of interaction such as voice control, gestures, gaze direction, etc. Once the user has provided feedback to the system, the next step consists of a detection / classification / location stage. In some cases, only detection is necessary, for example if the user wants to keep any speech occurring in the auditory scene. In other cases, classification may be necessary, for example if the user wants to keep fire alarms in the auditory scene, but not telephone ringing or office noises. In some cases, only the location of the source is relevant to the system. This is for example the case with the four sound sources in Fig. 9, where the user may want to remove or attenuate sound sources coming from a certain direction, regardless of the type or characteristics of the source. decision It is possible.

[0136] FIG. 10 illustrates a processing workflow for an SH application according to an embodiment.

[0137] First, the separation enhancement stage in Fig. 10 modifies the auditory scene. This is done by suppressing, attenuating, or enhancing a specific sound source (e.g., or a specific sound source). As shown in Fig. 10, an additional processing option in SH is noise control, which has the goal of removing or minimizing background noise in the auditory scene. Perhaps the most common and widespread technique for noise control is active noise control (ANC)

[11] .

[0138] Selective hearing is distinguished from virtual and augmented auditory environments by restricting selective hearing to applications where only real sound sources are modified in the auditory scene, without any attempt to add virtual sound sources to the scene.

[0139] From a machine hearing perspective, selective hearing applications require techniques to automatically detect, localize, classify, separate, and enhance sound sources. To further clarify the terminology related to selective hearing, the following terms are defined and their differences and relationships are highlighted.

[0140] In an embodiment, for example, source localization is used, which refers to the ability to detect the position of a sound source in an auditory scene. In the context of audio processing, source location usually refers to the direction of arrival (DOA) of a given sound source, which can be given either as a 2D coordinate (azimuth) or a 3D coordinate if it includes the elevation angle. Some systems also estimate the distance from the sound source to the microphone as position information [3]. In the context of music processing, location often refers to the panning of the sound source in the final mix, usually given as an angle in degrees [4].

[0141] According to an embodiment, for example, it is possible to determine whether any instances of a given sound source type are present in an auditory scene. decision Sound source detection is used to refer to the ability to detect the presence of a speaker in a scene. decision In this context, the number of speakers in a scene or the identity of the speakers is decision This is outside the scope of sound source detection, which can be understood as a binary classification task where the classes correspond to "source present" and "source absent".

[0142] In an embodiment, for example, sound source classification is used to assign a class label from a set of predefined classes to a given sound source or a given sound event. An example of a classification task is determining whether a given sound source corresponds to speech, music, or environmental noise. decision The goal of the proposed method is to detect the presence or absence of a source. Source classification and detection are closely related concepts. In some cases, classification systems include a detection stage by considering "no class" as one of the possible labels. In these cases, the system learns to detect the presence or absence of a source implicitly and is not forced to assign a class label if there is not enough evidence that one of the sources is active.

[0143] According to an embodiment, source separation is used with reference to, for example, the extraction of a given sound source from a sound mixture or an auditory scene. An example of source separation is the extraction of a singing voice from a sound mixture where, apart from the singer, other instruments are playing at the same time [5]. Source separation becomes relevant in selective listening scenarios, since it allows to suppress sound sources that are not of interest to the listener. Some sound separation systems implicitly perform a detection task before extracting the sound source from the mixture. However, this is not necessarily the rule, and therefore it is important to highlight the distinction between these tasks. Moreover, separation often serves as a pre-processing stage for other types of analysis, such as source enhancement [6] or classification [7].

[0144] In an embodiment, for example, audio source identification is used, which goes one step further and aims to identify specific instances of audio sources in an audio signal. Speaker identification is perhaps the most common use of audio source identification today. In this task, the goal is to identify whether a particular speaker is present in the scene. In the example of FIG. 1, the user has selected "Speaker X" as one of the audio sources held in the auditory scene. This requires techniques that go beyond voice detection and classification, and requires speaker-specific models that allow this accurate identification.

[0145] According to the embodiment, for example, source enhancement is used to refer to a process that increases the salience of a given source in an auditory scene [8]. For audio signals, the goal is often to increase its perceptual quality and intelligibility. A common scenario for speech enhancement is the denoising of speech corrupted by noise [9]. In the context of music processing, source enhancement is related to the concept of remixing and is often performed to make one instrument (source) more prominent in the mix. Remixing applications often use audio separation front-ends to access individual sources and modify the characteristics of the mixture

[10] . Although a source separation stage can be performed before source enhancement, this is not always the case, and therefore we also highlight the distinction between these terms.

[0146] In the field of sound source detection, classification, and identification, for example, some embodiments use one of the following concepts, such as acoustic scene and event detection and classification

[18] . In this context, a method has been proposed for audio event detection (AED) in a home environment, where the goal is to detect the time boundary of a given sound event within 10 seconds of recording

[19] ,

[20] . In this particular case, 10 sound event classes were considered, including cat, dog, talking, alarm, and tap water. Methods for polyphonic sound event (several simultaneous events) detection have also been proposed in

[21] ,

[22] . In

[21] , a method for polyphonic sound event detection is proposed, where a total of 61 sound events from a real context are detected using a binary activity detector based on a bi-directional long short-term memory (BLSTM) recurrent neural network (RNN).

[0147] For example, to handle weakly labeled data, some embodiments incorporate a temporal attention mechanism to focus on specific regions of the signal for classification

[23] . The problem of noisy labels in classification is particularly relevant for selective hearing applications where class labels are highly diverse and high-quality annotation is very costly

[24] . Noisy labels in sound event classification tasks are addressed in

[25] , where a noise-robust loss function based on categorical cross-entropy is presented, as well as a method to evaluate both noisy and manually labeled data. Similarly,

[26] presents a system for sound event classification based on a convolutional neural network (CNN) that incorporates a validation step of noisy labels based on the CNN's predicted consensus over multiple segments of training examples.

[0148] For example, some embodiments provide simultaneous detection and localization of sound events. Thus, some embodiments perform detection as a multi-label classification task, such as in

[27] , where the location is given as the 3D coordinates of the direction of arrival (DOA) of each sound event.

[0149] Some embodiments use concepts of voice activity detection and speaker recognition / identification for SH. Voice activity detection has been addressed in noisy environments using denoising autoencoders

[28] , recurrent neural networks

[29] , or as end-to-end systems using raw waveforms

[30] . For speaker recognition applications, numerous systems have been proposed in the literature

[31] , most of which focus on increasing robustness to different conditions, for example with data augmentation or improved embeddings that facilitate recognition

[32] -

[34] . Thus, some of the embodiments use these concepts.

[0150] Further embodiments use concepts for instrument classification for sound event detection. Instrument classification in both monophonic and polyphonic settings has been addressed in the literature

[35] ,

[36] . In

[35] , dominant instruments in 3-second audio segments are classified among 11 instrument classes and several aggregation techniques are proposed. Similarly,

[37] proposes a method for instrument activity detection that can detect instruments at a finer time resolution of 1 second. A significant amount of research has been carried out in the field of singing voice analysis. In particular, methods have been proposed such as

[38] for the task of detecting segments in recordings where singing is active. Some embodiments use these concepts.

[0151] Some of the embodiments use one of the concepts described below for sound source localization. Sound source localization is closely related to the problem of sound source counting, since the number of sound sources in an auditory scene is usually unknown in real applications. Some systems work under the assumption that the number of sources in a scene is known. This is the case, for example, of the model presented in

[39] , which uses a histogram of active intensity vectors to localize the sources. From a supervised perspective,

[40] proposes a CNN-based algorithm to estimate the DOA of multiple speakers in an auditory scene using a phase map as an input representation. In contrast, some works in the literature jointly estimate the number of sources in a scene and their location information. This is the case in

[41] , where a system for multi-speaker localization in noisy and reverberant environments is proposed. The system uses a complex-valued Gaussian Mixture Model (GMM) to estimate both the number of sources and their localization. The concepts described there are used by some of the embodiments.

[0152] Sound source localization algorithms can be computationally demanding, as they often involve scanning a large space around the auditory scene

[42] . To reduce the computational requirements in localization algorithms, some embodiments use concepts that reduce the search space by using clustering algorithms

[43] or by performing a multi-resolution search with established methods such as those based on the steered response power phase transform (SRP-PHAT)

[42] . Other methods impose sparsity constraints and assume that only one sound source dominates in a given time-frequency region

[44] . Recently, an end-to-end system for azimuth detection directly from raw waveforms has been proposed in

[45] . Some embodiments use these concepts.

[0153] Some embodiments use concepts described below for sound source separation (SSS), especially from the field of speech and music separation.

[0154] In particular, some embodiments use the concept of speaker-independent separation: separation is performed in situ without any prior information about the speakers in the scene

[46] . Some embodiments also estimate the spatial location of the speakers to perform separation

[47] .

[0155] Given the importance of computational performance in selective hearing applications, research conducted with the specific goal of achieving low latency is particularly important. Several studies have been proposed to perform low-latency speech separation (<10 ms) with little available training data

[48] . To avoid the delays caused by framing analysis in the frequency domain, some systems approach the separation problem by carefully designing filters applied in the time domain

[49] . Other systems achieve low-latency separation by directly modeling the time-domain signal using an encoder-decoder framework

[50] . In contrast, some systems attempt to reduce the framing delay in frequency-domain separation techniques

[51] . These concepts are adopted by some embodiments.

[0156] Some embodiments use concepts for music sound separation (MSS) to extract a musical source from a sound mixture [5], such as those for reed instrument accompaniment separation

[52] . These algorithms attempt to take the most prominent sound source in a mixture, regardless of its class label, and separate it from the remaining accompaniments. Some embodiments use concepts for singing voice separation

[53] . In most cases, either a specific source model

[54] or a data-driven model

[55] is used to capture the characteristics of singing voices. Although systems such as those proposed in

[55] do not explicitly incorporate a classification or detection stage to achieve separation, the data-driven nature of these techniques allows these systems to implicitly learn to detect singing voices with a certain accuracy before separation. Another class of algorithms in the music domain attempts to perform separation using only the location of the sound sources, without attempting to classify or detect the sources before separation [4].

[0157] Some embodiments use the concept of active noise control (ANC), such as active noise cancellation (ANC). ANC systems primarily aim to remove background noise for headphone users by introducing and canceling a noise-removing signal

[11] . ANC can be considered a special case of SH and faces similarly stringent performance requirements

[14] . Some studies have focused on active noise control in specific environments, such as automobile cabins

[56] or industrial scenarios

[57] . The work in

[56] analyzes the removal of different types of noise, such as road noise and engine noise, and requires a unified noise control system that can address different types of noise. Some studies have focused on developing ANC systems to remove noise over a specific spatial domain. In

[58] , ANC over a spatial domain is addressed using spherical harmonics as basis functions to represent the noise field. Some embodiments use the concepts described herein.

[0158] Some of the embodiments use the concept for audio source extension.

[0159] In the context of speech enhancement, one of the most common applications is the enhancement of speech corrupted by noise. Many studies have focused on phase processing for single-channel speech enhancement [8]. From the perspective of deep neural networks, the problem of speech denoising has been addressed in denoising autoencoders in

[59] , nonlinear regression problems between clean and noisy speech using deep neural networks (DNNs) in

[60] , and end-to-end systems using generative adversarial networks (GANs) in

[61] . Often, speech enhancement is applied as a front-end of automatic speech recognition (ASR) systems, as in the case of

[62] , which approaches speech enhancement with LSTM RNNs. Speech enhancement is also often performed in conjunction with source separation techniques, which may first extract speech and then apply enhancement techniques to the separated speech signal [6]. The concepts described herein are used by some embodiments.

[0160] In most cases, source enhancement in the context of music refers to applications for creating music remixes. In contrast to speech enhancement, where it is often assumed that speech is only impaired by noise sources, music applications almost always assume that other sound sources (musical instruments) are being played simultaneously with the source to be emphasized. For this reason, music remix applications are always provided to be preceded by a source separation stage. For example, in

[10] , early jazz recordings were remixed by applying lead accompaniment and harmonic percussion separation techniques to achieve a better tonal balance in the mixture. Similarly,

[63] studied the use of different singing voice separation algorithms to modify the relative volume of the singing voice and the backing track, showing that an increase of 6 dB was possible by introducing a slight but audible distortion in the final mixture. In

[64] , the authors study how to improve the music perception of cochlear implant users by applying source separation techniques to achieve a new mix. The concepts described there are used by some embodiments.

[0161] One of the biggest challenges in selective hearing applications relates to the stringent requirements on processing time. To maintain naturalness and perceived quality for the user, the complete processing workflow needs to be performed with minimal delay. The maximum tolerable latency of the system depends heavily on the application and the complexity of the auditory scene. For example, McPherson et al. propose 10 ms as an acceptable latency criterion for interactive music interfaces

[12] . For music performance over a network, the authors in

[13] report that latency becomes perceptible in the range of 20-25 ms to 50-60 ms. However, active noise control / cancellation (ANC) techniques require ultra-low latency processing for better performance. In these systems, the amount of tolerable latency depends on both frequency and attenuation, but can be as low as 1 ms for an attenuation of about 5 dB for frequencies below 200 Hz

[14] . A final consideration in SH applications refers to the perceived quality of the modified auditory scene. A significant amount of work has been devoted to methodologies for reliable assessment of speech quality in various applications

[15] ,

[16] ,

[17] . However, a challenge with SH is managing the clear trade-off between processing complexity and perceptual quality. Some embodiments use the concepts described therein.

[0162] Some embodiments use concepts for counting / counting and localization as described in

[41] , localization and detection as described in

[27] , separation and classification as described in

[65] , and separation and counting as described in

[66] .

[0163] Some embodiments use concepts to increase the robustness of current machine listening methods, as described in

[25] ,

[26] ,

[32] ,

[34] , and new emerging directions include domain adaptation

[67] and training on datasets recorded on multiple devices

[68] .

[0164] Some embodiments can handle raw waveforms and use concepts to improve the computational efficiency of machine hearing methods such as those described in

[48] or those described in

[30] ,

[45] ,

[50] ,

[61] .

[0165] Some embodiments provide an integrated optimization scheme that detects / classifies / localizes and separates / enhances in a combined manner so that sound sources in a scene can be selectively modified, while the independent detection, separation, localization, classification, and enhancement methods are reliable and provide the robustness and flexibility required for SH.

[0166] Some embodiments are suitable for real-time processing with a good trade-off between algorithmic complexity and performance.

[0167] Some embodiments combine ANC and machine hearing, for example, an auditory scene is first classified and then ANC is selectively applied.

[0168] Further embodiments are provided below.

[0169] In order to augment a real hearing environment with virtual sound objects, the transfer functions from each location of the sound object to each location of the listener in the room must be well known.

[0170] The transfer function maps the characteristics of the sound source and the direct sound between the object and the user, and all the reflections that occur in the room. To ensure correct spatial audio reproduction for the actual room acoustics that the listener is currently in, the transfer function must also map with sufficient accuracy the room acoustics characteristics in the listener's room.

[0171] In an audio system suitable for the representation of individual audio objects at different positions in a room, the challenge is proper detection and separation of the individual audio objects when multiple audio objects are present, and the audio signals of each object overlap at the recording or listening position in the room. The room acoustics and audio signal overlap change when the objects and / or listening positions in the room change.

[0172] Due to the relative motion, the estimation of the room acoustics parameters must be performed sufficiently quickly. Here, low estimation latency is more important than high accuracy. High accuracy is required when the positions of the sound sources and receivers do not change (static case). In the proposed system, the room acoustic parameters, as well as the room geometry and the listener positions, are estimated or extracted from a stream of audio signals. The audio signals are recorded in a real environment, where the sound source(s) and receiver(s) can move in any direction and the sound source(s) and / or receiver(s) can arbitrarily change their orientation.

[0173] The audio signal stream may be the result of any microphone setup, including one or more microphones. The stream is fed to a signal processing stage for pre-processing and / or further analysis. The output is then fed to a feature extraction stage. This stage estimates room acoustic parameters, e.g. T60 (reverberation time), DRR (Direct-to-Reverberant Ratio), etc.

[0174] The second data stream is generated by a 6DoF sensor ("6 degrees of freedom": three dimensions of room position and gaze direction) that captures the orientation and position of the microphone setup. The position data stream is fed into a 6DoF signal processing stage for pre-processing or further analysis.

[0175] The output of the 6DoF signal processing, the audio feature extraction stage, and the preprocessed microphone streams are fed into a machine learning block, where the auditory space, i.e. the inside of the listening room (size, geometry, reflective surfaces), and the position of the microphone field within the room are estimated. Furthermore, to enable more robust estimation, a user behavior model is applied. This model takes into account the limitations of human motion (e.g. continuous movement, speed, etc.), as well as the probability distribution of different types of motion.

[0176] Some of the embodiments use arbitrary microphone arrangements and achieve blind estimation of room acoustic parameters by adding user position and pose information, and by analyzing the data with machine learning methods.

[0177] For example, systems according to embodiments can be used for acoustically augmented reality (AAR), where a virtual room impulse response has to be synthesized from estimated parameters.

[0178] Some embodiments include the removal of reverberation from the recorded signal. An example of such an embodiment is a hearing aid for normal hearing people and for people with hearing loss. In this case, with the help of the estimated parameters, reverberation can be removed from the input signal of the microphone setting.

[0179] A further application is the spatial synthesis of sound scenes generated in rooms other than the current auditory space: for this purpose the room acoustics parameters which are part of the sound scene are adapted to the room acoustics parameters of the auditory space.

[0180] In the case of binaural synthesis, for this purpose the available BRIRs are adapted to the different acoustic parameters of the auditory space.

[0181] In one embodiment, one or more room acoustics parameters are decision An apparatus for performing the method is provided.

[0182] The device is configured to acquire microphone data including one or more microphone signals.

[0183] Additionally, the device is configured to obtain tracking data regarding the position and / or orientation of the user.

[0184] Additionally, the device may calculate one or more room acoustics parameters in response to the microphone data and the tracking data. decision The device is configured to:

[0185] According to one embodiment, for example, the device may determine one or more room acoustics parameters in response to the microphone data and the tracking data. decision The device may be configured to use machine learning to perform the classification.

[0186] In an embodiment, for example, the device may be configured to use machine learning, in that the device may be configured to use neural networks.

[0187] According to one embodiment, for example, the device may be configured to use cloud-based processing for machine learning.

[0188] In one embodiment, for example, the one or more room acoustics parameters may include reverberation time.

[0189] According to one embodiment, for example, the one or more room acoustics parameters may include a directivity to reverberation ratio.

[0190] In one embodiment, for example, the tracking data may include x, y, and z coordinates to label the user's location.

[0191] According to an embodiment, for example, tracking data may include pitch, yaw, and roll coordinates for labeling the user's orientation.

[0192] In an embodiment, for example, the device may be configured to transform one or more microphone signals from the time domain to the frequency domain, for example, the device may be configured to extract one or more features of the one or more microphone signals in the frequency domain, and the device may calculate one or more room acoustic parameters in response to the one or more features. decision The device may be configured to:

[0193] According to one embodiment, for example, the device may be configured to use cloud-based processing to extract one or more features.

[0194] In one embodiment, for example, the device may include a microphone arrangement of several microphones for recording several microphone signals.

[0195] According to one embodiment, for example, the microphone arrangement may be configured to be worn on the body of a user.

[0196] In an embodiment, for example, the system described above may include: decision The device may further include the above-mentioned apparatus for

[0197] According to one embodiment, for example, the signal portion modifier 140 and / or the signal generator 150 may be configured to perform the generation of at least one of the plurality of binaural room impulse responses for each of the one or more sound sources depending on the at least one of the one or more room acoustics parameters.

[0198] FIG. 7 shows a system according to one embodiment, which includes five subsystems (Subsystems 1-5).

[0199] Subsystem 1 includes a microphone setup of one, two or more individual microphones, which can be combined into a microphone field if one or more microphones are available. The positioning and relative placement of the microphones / microphones to each other can be arbitrary. The microphone configuration can be part of a device worn by the user or can be a separate device placed in the room of interest.

[0200] Subsystem 1 also includes a tracking device that measures the user's translational position in the room and the user's head pose, with up to 6DoF (x, y, z, pitch, yaw, and roll angles) being measured.

[0201] The tracking device may be placed on the user's head or may be split into several sub-devices to measure the required DoF, and may or may not be placed on the user.

[0202] Thus, subsystem 1 represents an input interface including a microphone signal input interface 101 and a position information input interface 102 .

[0203] Subsystem 2 includes signal processing of the recorded microphone signals. This includes frequency conversion and / or time domain based processing. Furthermore, it includes methods for combining different microphone signals to realize field processing. Feedback from system 4 is possible to adapt the parameters of the signal processing in subsystem 2. The signal processing blocks of the microphone signals may be part of the device in which the microphones are embedded or may be part of a separate device. It may also be part of cloud-based processing.

[0204] Furthermore, subsystem 2 includes signal processing for the recorded tracking data. This includes frequency transformation and / or time domain based processing. Furthermore, it includes methods for improving the technical quality of the signal by using noise suppression, smoothing, interpolation, and extrapolation. Furthermore, it includes methods for deriving higher level information. This includes speed, acceleration, path direction, idle time, movement range, and movement path. It also includes predictions of near future movement path and near future speed. The signal processing block of the tracking signal may be part of the tracking device or may be part of a separate device. It may also be part of cloud-based processing.

[0205] Subsystem 3 contains the processed microphone feature extraction.

[0206] The feature extraction block may be part of the user's wearable device, or it may be part of a separate device, or it may be part of a cloud-based process.

[0207] Subsystems 2 and 3, together with their modules 111 and 121, implement, for example, detector 110, voice type classifier 130, and signal portion modifier 140. For example, subsystem 3, module 121 can output the result of voice classification to subsystem 2, module 111 (feedback). For example, subsystem 2, module 112 can output the result of voice classification to subsystem 2, module 112 (feedback). decision Additionally, in one embodiment, subsystems 2 and 3 may also realize a signal generator 150, e.g., by subsystem 2, module 111 generating the binaural room impulse response and loudspeaker signals.

[0208] Subsystem 4 includes methods and algorithms to estimate room acoustic parameters using the processed microphone signals, extracted microphone signal features, and processed tracking data. The output of this block is the room acoustic characteristic parameters as idle data, as well as control and variation of parameters of microphone signal processing in subsystem 2. Machine learning block 131 may be part of the user's device or a separate device. It may also be part of cloud-based processing.

[0209] Additionally, Subsystem 4 includes post-processing of room acoustic idle data parameters (e.g., in block 132). This includes outlier detection, combining individual parameters into new parameters, smoothing, extrapolation, interpolation, and validation. This block also gets information from Subsystem 2, including the near future position of the user in the room, to estimate the near future acoustic parameters. This block may be part of the user's device or a separate device. It may also be part of cloud-based processing.

[0210] Subsystem 5 includes storage and allocation of room acoustics parameters for downstream systems (e.g., in memory 141). Parameter allocation may be accomplished just-in-time and / or time-responsively stored. Storage may be performed in a device located on or near the user, or in a cloud-based system.

[0211] Use cases for embodiments of the present invention are described below.

[0212] An embodiment use case is home entertainment, involving a user in a home environment. For example, a user wants to focus on a particular playback device, such as a TV, radio, PC, tablet, etc., and wants to suppress other sources of disturbance (other users' or children's devices, construction noise, street noise). In this case, the user positions himself close to the preferred playback device and selects the device or its location. Regardless of the user's location, the selected device, or sound source location, is acoustically emphasized until the user cancels his selection.

[0213] For example, a user moves closer to a target sound source, the user selects the target sound source via an appropriate interface, and the hearable accordingly adapts the sound playback based on the user position, the user's gaze direction, and the target sound source, so that the target sound source can be fully understood even in the presence of noise interference.

[0214] Alternatively, the user moves near a particularly disruptive sound source, which the user selects via an appropriate interface and the hearable adjusts the audio playback accordingly based on the user's position, the user's gaze direction, and the disruptive sound source to explicitly accommodate the disruptive sound source.

[0215] A further use case for a further embodiment is a cocktail party where the users are positioned between several speakers.

[0216] In the presence of many talkers, for example, the user may wish to focus on one (or more) of them and tune out or attenuate other sources of disturbance. In this use case, control of the hearable should require little interaction from the user. Control of the strength of selectivity based on biosignals or detectable indicators of speech difficulty (frequent questions, foreign languages, strong diamond words) would be optional.

[0217] For example, speakers are randomly distributed and move relative to the listener. In addition, there are periodic pauses in speech, new speakers are added or others leave the scene. Perhaps external disturbing sounds, such as music, are relatively loud. The selected speaker is acoustically emphasized and is recognized again after a pause in speech, a change in his or her position or posture.

[0218] For example, hearables recognize speakers in the user's vicinity. With appropriate controllability (e.g., gaze direction, attention control), the user can select a preferred speaker. By adapting the audio playback according to the user's gaze direction and the selected target sound source, the hearables can fully understand the target sound source even in the presence of noise interference.

[0219] Alternatively, if a user is directly addressed by a (previously) non-preferred speaker, the user must at least be audible to ensure natural communication.

[0220] Another use case of another embodiment is in an automobile where the user is located in his / her (or her) car. While driving, the user wants to actively direct his / her acoustic attention to certain playback devices such as a navigation device, a radio, or a conversation partner so that he / she can better understand them next to the interfering noises (wind, motor, passengers).

[0221] For example, the user and the target sound source are placed at fixed locations in a car. The user is stationary relative to the reference system, but the vehicle itself is moving. This requires an adaptive tracking solution. The selected sound source location is acoustically highlighted until the user cancels the selection or a warning signal stops the device functioning.

[0222] For example, a user gets into a car and the surroundings are detected by the device. With appropriate controllability (e.g., speed awareness), the user can switch between target sound sources, and the hearable adapts the sound playback according to the user's gaze direction and the selected target sound source, so that the target sound source can be fully understood even in the presence of noise interference.

[0223] Alternatively, for example, a traffic-related warning signal interrupts the normal flow and cancels the user's selection, after which resumption of normal flow is performed.

[0224] Another use case of further embodiments is live music and concerns guests at live music events. For example, guests at a concert or live music performance may wish to increase their focus on the performance with the help of those who can hear, and wish to ignore other guests who behave in a disruptive manner. Furthermore, the audio signal itself may be optimized, for example to balance unfavourable listening positions or room acoustic effects.

[0225] For example, the user is located among many sources of disturbance. In most cases, however, the performance is relatively large. The target sound sources are located at fixed positions or at least in a predefined area, while the user is highly mobile (e.g., the user may be dancing). The selected sound source position is acoustically emphasized until the user cancels the selection or until a warning signal stops the device functioning.

[0226] For example, the user selects a stage area or musicians as the target sound source. With appropriate controllability, the user can define the stage / musician location and the hearable adapts the sound playback according to the user's gaze direction and the selected target sound source so that the target sound source can be fully understood even in the case of noise interference.

[0227] Alternatively, for example, warning information (e.g., evacuation in case of outdoor events, approaching thunderstorms) and warning signals can interrupt the normal flow and cancel the user's selection, after which there is a resumption of normal flow.

[0228] A further use case of another embodiment is for major events and guests at major events, so hearables can be used at major events (e.g. football stadiums, ice hockey stadiums, large concert halls, etc.) to highlight the voices of family and friends that would otherwise be drowned out by crowd noise.

[0229] For example, a large event with many attendees takes place in a stadium or large concert hall. A group (family, friends, school class) attends the event and is located outside or inside the event location where a large crowd roams. One or more children lose eye contact with the group and seek out the group despite high noise levels caused by noise. The user then turns off the voice recognition and the hearable no longer amplifies the sound.

[0230] For example, a person in the group can select the lost child's voice on the hearable, the hearable will find the voice, the hearable will amplify the voice and the user can recover the lost child (faster) based on the amplified voice.

[0231] Alternatively, for example, a missing child also wears a hearable and selects a parent's voice. The hearable amplifies the parent's voice. The amplification allows the child to locate the parents. This allows the child to walk back to the parent. Alternatively, for example, a missing child also wears a hearable and selects a parent's voice. The hearable finds the parent's voice(s) and announces the distance to the voice(s). This helps the child to find the parent. Optionally, playback of an artificial voice from the hearable may be provided for distance announcement.

[0232] For example, coupling of hearables for selective amplification of sound is provided and sound profiles are stored.

[0233] A further use case of a further embodiment is recreational sports and concerns recreational athletes. Listening to music during sports is popular; however, it also involves dangers. Warning signals or other road users may not be heard. In addition to playing music, the hearables can react to warning signals or voices and temporarily interrupt the music playback. In this context, a further use case is sports in small groups. The hearables of a sports group can be connected to ensure good communication during sports while suppressing other disturbing noises.

[0234] For example, users can be mobile and possible warning signals are overlapped by many sources of disturbance. The problem is that not all warning signals are potentially relevant to the user (remote sirens in the city, horns in the street). This causes the hearable to automatically stop music playback and acoustically emphasize the communication partner's warning signal until the user cancels the selection. The music then plays normally.

[0235] For example, a user is engaged in sports and listening to music via a hearable. An alert signal or voice regarding the user is automatically detected and the hearable pauses the music playback. The hearable adapts the audio playback to fully understand the target sound source / acoustic environment. The hearable then continues playing the music automatically (e.g., after the alert signal ends) or as requested by the user.

[0236] Or, for example, athletes in a group can connect their hearables, optimizing speech intelligibility between group members and suppressing other interfering noises.

[0237] Another use case of another embodiment is the suppression of snoring, which concerns all those who want to sleep that is disturbed by snoring. Those whose partners are snoring have trouble sleeping because their night rest is disturbed. The hearable provides a sense of security, since it suppresses the snoring sounds, guarantees a night's rest and provides security in the home. At the same time, the hearable lets through other sounds (babies screaming, alarm sounds, etc.) so that the user is not completely acoustically isolated from the outside world. For example, snore detection is provided.

[0238] For example, a user has a sleep disorder due to snoring sounds. By using the hearable, the user can sleep better again, which has a stress reducing effect.

[0239] For example, a user wears the hearable while sleeping. He / she switches the hearable to a sleep mode that suppresses all snoring sounds. After going to sleep, he / she turns the hearable off again.

[0240] Alternatively, it can suppress other sounds such as construction noise, lawnmower noise, etc. while you sleep.

[0241] A further use case of a further embodiment is a diagnostic device for the user in daily life. The hearable records preferences (e.g. which sound source is selected, which attenuation / amplification is selected) and creates a profile with trends over the period of use. This data may make it possible to draw conclusions regarding changes in hearing ability. The aim is to detect hearing loss as soon as possible.

[0242] For example, a user carries the device with them in their daily life or the mentioned use cases for months or years, and the hearable creates analytics based on the selected settings and outputs warnings and recommendations to the user.

[0243] For example, a user wears a hearable for an extended period of time (months to years), the device creates an analysis based on the hearing preferences, and the device outputs recommendations and warnings in case of the onset of hearing loss.

[0244] A further use case of another embodiment is as a therapeutic device, and concerns users with hearing impairment in their daily life. In its role as a transitional device on the way to a hearing device, potential patients are assisted as early as possible, thus treating dementia preventatively. Other possibilities are its use as a concentration trainer (e.g. in the case of ADHS), for the treatment of tinnitus, and for stress reduction.

[0245] For example, a listener may have a hearing problem or attention deficit and temporarily / provisionally use a hearable as a hearing device. Depending on the hearing problem, it may be alleviated by the hearing device, for example, by amplification of all signals (hardness of hearing), high selectivity of preferred sound sources (attention deficit), playback of therapeutic sounds (treatment of tinnitus).

[0246] The user, independently or based on a physician's advice, selects the form of treatment and makes the desired adjustments, and the hearable carries out the selected treatment.

[0247] Alternatively, the hearable may detect hearing problems from the UC-PRO1 and automatically adapt playback based on the detected problem and notify the user.

[0248] A further use case of a further embodiment is public sector work and concerns public sector employees, who are subject to high levels of noise during their work (hospitals, pediatricians, airport counters, educators, restaurant industry, service counters, etc.) wear hearables for better communication of one or a few people's speech with emphasis, for example through stress reduction, and for better safety at work.

[0249] For example, employees experience high levels of noise in their work environment and need to talk to clients, patients, or coworkers despite the background noise without being able to switch to a quieter environment. Hospital employees experience high levels of noise from medical devices ringing and beeping (or any other work-related noise) and must still be able to communicate with patients or coworkers. Pediatricians and educators must work amidst noise or screaming children and be able to talk to parents. At airport counters, employees have difficulty understanding airline passengers when the noise levels are high in the airport concourse. Waiters have difficulty hearing customers' orders amidst the noise of a frequented restaurant. Then, for example, the user turns off the voice selection and the hearable no longer amplifies the voice.

[0250] For example, a person turns on an equipped hearable, the user sets the hearable to audio selection for nearby voices, and the hearable amplifies the nearby voice, or a small number of nearby voices, while simultaneously suppressing background noise, so that the user can better understand the relevant voices.

[0251] Alternatively, a person sets the hearable to continuous noise suppression. The user turns on the function to detect and amplify available sounds. Thus, the user can continue working with a lower level of noise. When directly addressed from within x meters, the hearable then amplifies the sound. Thus, the user can converse with others with a lower level of noise. After the conversation, the hearable goes back to noise suppression mode, and after work, the user turns the hearable off again.

[0252] Another use case of another embodiment is passenger transportation, which concerns users of automobiles for the transportation of passengers. For example, users and drivers of passenger transport vehicles want their passengers to be as distracted as possible while driving. Even though passengers are a major source of disturbance, communication with them is necessary from time to time.

[0253] For example, the user or driver and the disturbance source are located at fixed positions in a car. The user is stationary relative to the reference system, but the vehicle itself is moving. This requires an adaptive tracking solution. Thus, passenger sounds and conversations are acoustically suppressed by default unless communication is taking place.

[0254] For example, the hearable will suppress noise for the occupants by default. The user can manually override the suppression through appropriate control possibilities (voice recognition, buttons in the vehicle), where the hearable will adapt the audio playback according to the selection.

[0255] Alternatively, the hearable may detect when an occupant is actively talking to the driver and temporarily suspend noise suppression.

[0256] Another use case of further embodiments is school and education, involving teachers and students in a class. In one example, the hearable has two roles and the functions of the devices are partially combined. The teacher / speaker's device suppresses interfering noise and amplifies the speech / questions from the students. The hearable of the listener may also be controlled via the teacher's device. Thus, particularly important content can be highlighted without the need to speak louder. Students can set their hearables in such a way that they can understand the teacher better and filter out disruptive classmates.

[0257] For example, the teacher and students are located in defined areas in a closed space (this is the rule). If all devices are coupled to each other, the relative positions are interchangeable, which simplifies source separation. The selected sound source is acoustically emphasized until the user (teacher / student) cancels the selection or until a warning signal interrupts the functioning of the device.

[0258] For example, a teacher or speaker presents content and the device suppresses disturbing noise. The teacher wants to hear a student's question and changes the focus of the hearable (automatically or via suitable control possibilities) to the person with the question. After the communication, all sounds are suppressed again. Furthermore, it may be provided that students who feel, for example, that they are being disturbed by their classmates, tune them out acoustically. For example, a student sitting far away from the teacher may have the teacher's voice amplified.

[0259] Alternatively, for example, teacher and student devices may be combined. Selectivity of student devices may be temporarily controlled via the teacher device. For particularly important content, the teacher changes selectivity of student devices to amplify his or her voice.

[0260] A further use case of another embodiment is military and concerns soldiers. On the one hand, verbal communication between soldiers in the field is done via radio, on the other hand, via voice and direct contact. Radio is mostly used when communication is between different units and subgroups. A certain radio etiquette is often used. Ejection and direct contact are mostly done to communicate within a unit or group. During a soldier's mission, difficult acoustic conditions (e.g. people screaming, weapon noise, bad weather) may exist that can impair both communication paths. Radio devices with earphones are often part of the soldier's equipment. In addition to the purpose of voice reproduction, they also provide a protection function against higher levels of sound pressure. These devices are often equipped with a microphone to bring environmental signals to the carrier's ear. Active noise suppression is also part of such systems. Enhancement / extension of the functional range allows the soldier's voice-ejection and direct contact in noisy environments by intelligent attenuation of interference noise and selective emphasis of voice with directional reproduction. For this purpose, the relative position of the soldier in the room / field must be known. Furthermore, the voice signal and the interfering noise must be separated from each other spatially and by content. The system must be able to handle high SNR levels as well, from low whispers to screams and explosions. The advantages of such a system are: verbal communication between soldiers in noisy environments, preservation of hearing protection, possibility of waiving radio etiquette, and (as it is not a radio solution) interception security.

[0261] For example, voice and direct contact between soldiers on a mission can be complicated by interference noise. This problem is currently being addressed by short-range and longer-range wireless solutions. New systems enable voice and direct contact in the near field with intelligent spatial emphasis of each speaker and attenuation of ambient noise.

[0262] For example, a soldier is on duty. Voices and sounds are automatically detected and the system amplifies them with simultaneous attenuation of background noise. The system adapts the spatial audio playback to ensure that the target sound source is fully understood.

[0263] Or, for example, the system can know which soldiers are in a group and only pass through the voice signals of those group members.

[0264] A further use case of a further embodiment concerns security guards and guards. Thus, for example, hearables can be used to disrupt major events (celebrations, demonstrations) for preemptive detection of crimes. The selectivity of the hearable is controlled by keywords, for example calls for help or calls for violence. This presupposes a content analysis of the audio signal (for example voice recognition).

[0265] For example, a security guard is surrounded by many loud sound sources, and the guard and all sound sources may be moving. A person calling for help may not be able to hear under normal hearing conditions or can only hear to a limited extent (bad SNR). A manually or automatically selected sound source is acoustically emphasized until the user cancels the selection. Optionally, a virtual sound object is placed at the location / direction of the sound source of interest to make the location (e.g. in case of a one-off call for help) easy to find.

[0266] For example, the hearable detects a sound source that has a potential danger source. The guard selects (e.g., by selection on a tablet) which sound source or event he wants to follow. The hearable then adapts the audio playback so that the sound source can be well understood and located even in the presence of noise interference.

[0267] Alternatively, for example, if the target sound source is silent, a localization signal may be positioned towards / within the distance of the sound source.

[0268] Another use case of another embodiment is communication on stage, related to musicians. On stage, in a rehearsal or concert (e.g. band, orchestra, chorus, music), a single instrument (group) may not be heard due to difficult acoustic conditions, even if it is still heard in other environments. This impairs the dialogue, since important (accompanying) sounds are no longer perceptible. Hearables can highlight these sounds and make them hearable again, thus improving or guaranteeing the dialogue of the individual musicians. This use can reduce the noise exposure of the individual musicians, preventing hearing loss, for example by attenuating the drums, and also allows the musicians to hear all important things at the same time.

[0269] For example, a musician without hearables can no longer hear at least one other voice on stage. In this case, hearables may be used. After the rehearsal or concert is over, the user turns off and then removes the hearables.

[0270] In one example, a user turns on the hearable. The user selects one or more desired instruments to be amplified. When creating music together, the selected musical instruments are amplified and thus made audible again by the hearable. After creating the music, the user turns off the hearable again.

[0271] In another example, a user turns on a hearable. The user selects a desired instrument that they would like to have the volume reduced. When creating music together, the volume of the selected instrument is reduced by the hearable so that the user can only hear it at a moderate volume.

[0272] For example, an instrument profile can be stored in a hearable.

[0273] Another use case of a further embodiment is source separation as a software module for hearing devices in the sense of an ecosystem, which concerns the manufacturer of the hearing device, or the user of the hearing device. The manufacturer can use source separation as an additional tool for the hearing device and offer it to the customer. Thus, the hearing device can also benefit from the development. Licensing models for other markets / devices (headphones, mobile phones, etc.) are also conceivable.

[0274] For example, a user of a hearing device has difficulty separating different sound sources in a complex hearing situation, for example to focus on a particular speaker. In order to be able to selectively listen without an external additional system (e.g., transfer of signals from a mobile radio set via Bluetooth, selective transfer of signals in a classroom via FM equipment or inductive hearing equipment), the user uses a hearing device with an additional function for selective listening. Thus, without any external effort, the user can focus on individual sound sources through sound source separation. Finally, the user turns off the additional function and continues to listen normally with the hearing device.

[0275] For example, a hearing device user acquires a new hearing device with an integrated additional feature for selective hearing. The user sets up the feature for selective hearing in the hearing device. The user then selects a profile (e.g. amplify the loudest / closest sources, amplify speech recognition of specific sounds in the personal surroundings (e.g. UC-CE5-Major Events, etc.)). The hearing device amplifies the respective sound source according to the set profile, simultaneously suppressing background noise on request, and the user of the hearing device hears individual sound sources from a complex hearing scene and not just the "noise" / clutter of sound sources.

[0276] Or the user of the hearing device gets an add-on feature for selective hearing, e.g. as software for his hearing device. The user installs the add-on feature on his hearing device. Then the user configures the feature for selective hearing on the hearing device. The user selects a profile (amplify the loudest / closest sound sources and amplify speech recognition of specific sounds from the personal surroundings (e.g. UC-CE5-Major Events)) and the hearing device amplifies the respective sound source according to the configured profile, while simultaneously suppressing background noise on request. In this case the user of the hearing device hears the individual sound sources from the complex hearing scene, not just the clutter of "noise" / sound sources.

[0277] For example, a hearable can provide a memorizable audio profile.

[0278] A further use case of further embodiments concerns professional sports and athletes in competitions. In sports such as biathlon, triathlon, cycling, marathon, etc., professional athletes rely on coaches' information or communication with teammates. However, there are situations where they want to protect themselves from loud sounds (shooting in biathlon, loud applause, party horns, etc.) to be able to concentrate. The hearables can be adapted to the respective sport / athlete to allow a fully automatic selection of the relevant sound source (detection of specific voices, volume limiting for typical disturbing noises).

[0279] For example, users are very mobile and the type of disturbing noise depends on the sport. Due to intense physical tension, control of the device by the athlete is not possible or only to a limited extent. However, in most sports there are prescribed procedures (biathlon: running, shooting) and important communication partners (trainer, teammates) can be predefined. Noise is suppressed in general or at certain stages of the activity. Communication between the athlete and teammates and coaches is always emphasized.

[0280] For example, athletes use hearables that are specially tuned for their type of sport. The hearables fully automatically (pretuned) suppress disturbing noises, especially in situations that require high levels of attention in each type of sport. In addition, the hearables fully automatically (pretuned) highlight trainers and team members when they are within hearing range.

[0281] A further use case of further embodiments is hearing training, for music students, professional musicians, and hobby musicians. In music rehearsals (e.g., in orchestras, bands, ensembles, music classes), hearables can be used selectively to track individual voices in a filtered way. Especially at the beginning of a rehearsal, it is useful to listen to the final recording of a piece and track your own voice. In some compositions, hearing foreground voices alone does not allow you to hear background voices well. Hearables can selectively highlight voices based on instruments, etc., for more targeted practice.

[0282] Music students (who wish to do so) could use hearables to train their hearing abilities and prepare for exams selectively by gradually minimizing individual emphasis until they could finally extract individual sounds from complex pieces without aid.

[0283] A further possible use case is karaoke, for example when Singstar etc. is not available nearby: vocals can be suppressed from the music if needed, to hear only the instrumental version for signing karaoke.

[0284] For example, a musician may begin learning the sounds from a song by listening to a recording of the music via a playback medium such as a CD player. When the user is done practicing, they turn off the hearables again.

[0285] In one example, a user turns on the hearable. Selects the desired instrument they want to amplify. When listening to music, the hearable amplifies the sound of the musical instrument and reduces the volume of the remaining musical instruments, thus allowing the user to better track their own voice.

[0286] In another example, a user turns on the hearable; selects the desired instrument they wish to suppress; and when listening to a song, the audio of the selected song is suppressed so that only the remaining audio is heard. Thus, the user can practice audio on their instrument with other audio without being distracted by audio from the recording.

[0287] In an embodiment, the hearable can provide a stored instrument profile.

[0288] Another use case of another embodiment is safety at work, which concerns workers in noisy environments: Workers in noisy environments such as machine halls or construction sites must protect themselves from the noise, but must also be able to perceive warning signals and communicate with their colleagues.

[0289] For example, the user may be located in a very loud environment and the desired sound source (warning signal, colleague) may be significantly softer than the disturbing noise. The user may be mobile. However, the disturbing noise is often stationary. Similar to hearing protection, the noise is permanently reduced and the hearable accentuates the warning signal completely automatically. Communication with the colleague is ensured by the amplification of the loudspeaker sound source.

[0290] For example, a user is at work and uses a hearable as hearing protection: a warning signal (e.g., a fire alarm) is acoustically accentuated, causing the user to stop working if necessary.

[0291] Or, for example, the user is at work and uses the hearable as hearing protection, and when there is a need to communicate with a colleague, a communication partner is selected and acoustically enhanced with the help of a suitable interface (here, for example, eye control).

[0292] Another use case of a further embodiment is source separation as a software module for a live translator, with respect to the user of the live translator. The live translator translates spoken foreign languages ​​in real time and can benefit from an upstream software module for source separation. In particular, in the case of multiple speakers, the software module can extract the target speaker and potentially improve the translation.

[0293] For example, the software module is part of a live translator (a dedicated device or an app on a smartphone). For example, the user can select a target speaker via the device's display. Advantageously, the user and the target sound source do not move or move only slightly during the translation. The selected sound source position is acoustically emphasized, thus potentially improving the translation.

[0294] For example, a user may wish to converse in a foreign language or listen to a speaker of a foreign language. The user selects the target speaker via an appropriate interface (e.g., a GUI on the display) and the software module optimizes the recording for further use in the translator.

[0295] A further use case of another embodiment is safety in the work of rescue teams, for firefighters, civil protection, police, emergency services. For the mitigating forces, good communication is essential to successfully handle the mission. Despite the high ambient noise, hearing protection is often not possible, since the relief makes communication impossible. For example, firefighters must be able to accurately transmit orders and understand them, despite the loud motor sounds that occur in part through the radio. The relief forces are therefore exposed to loud noises that do not allow compliance with hearing protection regulations. On the one hand, the hearable provides hearing protection for the relief forces, and on the other hand still allows communication between the relief forces. Furthermore, with the help of the hearable, the relief forces are not acoustically isolated from the environment when carrying the helmet / protective equipment and can therefore provide better support. They can communicate better and also estimate their own danger better (for example, hear the type of fire that is occurring).

[0296] For example, a user is subjected to strong ambient noise and therefore cannot wear hearing protection and must still be able to communicate with others. Use the hearable. After the mission is completed or the dangerous situation has ended, the user removes the hearable again.

[0297] For example, a user wears a hearable during a mission. The hearable is turned on. The hearable suppresses background noise and amplifies the speech of nearby colleagues and other speakers (e.g., fire victims).

[0298] Alternatively, a user wears the hearable while on a mission, turns the hearable on, and the hearable suppresses background noise and amplifies the voice of their colleagues over the radio.

[0299] Where applicable, the hearable is specially designed to meet structural compliance for operation according to operational specifications. In some cases, the hearable includes an interface to a wireless device.

[0300] While some aspects are described in the context of a device, it will be understood that said aspects also represent a description of a corresponding method, such that blocks or structural components of the device should also be understood as corresponding method steps or features of method steps. Similarly, aspects described in the context of or as method steps also represent a description of corresponding blocks or details or features of the corresponding device. Some or all of the method steps may be performed using hardware devices such as microprocessors, programmable computers, or electronic circuits. In some embodiments, some or some of the most important method steps may be performed by such devices.

[0301] Depending on the specific implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, a DVD, a Blu-ray disk, a CD, a ROM, a PROM, an EPROM, an EEPROM or a flash memory, a hard disk, or any other magnetic or optical memory on which electronically readable control signals are stored that cooperate or can cooperate with a programmable computer system so that the respective method is executed. This is why the digital storage medium can be computer readable.

[0302] Thus, some embodiments according to the present invention include a data carrier comprising electronically readable control signals capable of cooperating with a programmable computer system to cause any of the methods described herein to be performed.

[0303] Generally, embodiments of the present invention may be implemented as a computer program product having program code which is effective to perform any of the methods when the computer program product is run on a computer.

[0304] The program code may for example be stored on a machine readable carrier.

[0305] Other embodiments comprise the computer program for performing any of the methods described herein, said computer program stored on a machine readable carrier. In other words, an embodiment of the inventive method is therefore a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

[0306] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer readable medium) having recorded thereon a computer program for performing any of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible or non-volatile.

[0307] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing any of the methods described herein. The data stream or signal sequence can for example be adapted to be transmitted via a data communications link, for example via the Internet.

[0308] Further embodiments comprise a processing unit, e.g. a computer or a programmable logic device, configured or adapted to perform any of the methods described herein.

[0309] A further embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0310] Further embodiments according to the invention include a device or system configured to transmit a computer program for performing at least one of the methods described herein to a receiver. The transmission may be, for example, electronic or optical. The receiver may be, for example, a computer, a mobile device, a memory device, or a similar device. The device or system may, for example, include a file server for transmitting the computer program to the receiver.

[0311] In some embodiments, a programmable logic device (e.g., a field programmable gate array, FPGA) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are performed in some embodiments by any hardware device. The hardware device may be any universally applicable hardware, such as a computer processor (CPU), or may be method-specific hardware, such as an ASIC.

[0312] The above-described embodiments merely represent illustrative of the principles of the present invention. It is understood that others skilled in the art will appreciate modifications and variations of the arrangements and details described herein. As such, it is intended that the present invention be limited only by the scope of the following claims and not by the specific details presented herein by way of the description and discussion of the embodiments.

[0313] References [1] V.Valimaki,A.Franck,J.Ramo,H.Gamper,and L.Savioja,”Assisted listening using a headset:Enhancing audio perception in real,augmented,and virtual environments,” IEEE Signal Processing Magazine,volume 32,no.2,pp.92-99,March 2015

[0314] [2] K.Brandenburg,E.Cano,F.Klein,T.Kollmer,H.Lukashevich,A.Neidhardt,U.Sloma,and S.Werner,”Plausible augmentation of auditory scenes using dynamic binaural synthesis for personalized auditory realities,” in Proc.of AES International Conference on Audio for Virtual and Augmented Reality,August 2018

[0315] [3] S.Argentieri,P.Dans,and P.Soures,”A survey on sound source localization in robotics:From binaural to array processing methods,” Computer Speech Language,volume 34,no.1,pp.87-112,2015

[0316] [4] D.FitzGerald,A.Liutkus,and R.Badeau,”Projection-based demixing of spatial audio,” IEEE / ACM Trans.on Audio,Speech,and Language Processing,volume 24,no.9,pp.1560-1572,2016

[0317] [5] E.Cano,D.FitzGerald,A.Liutkus,M.D.Plumbley,and F.Stoter,”Musical source separation:An introduction,” IEEE Signal Processing Magazine,volume 36,no.1,pp.31-40,January 2019

[0318] [6] S.Gannot,E.Vincent,S.Markovich-Golan,and A.Ozerov,”A consolidated perspective on multimicrophone speech enhancement and source separation,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 25,no.4,pp.692-730,April 2017

[0319] [7] E.Cano,J.Nowak,and S.Grollmisch,”Exploring sound source separation for acoustic condition monitoring in industrial scenarios,” in Proc.of 25th European Signal Processing Conference(EUSIPCO),August 2017,pp.2264-2268

[0320] [8] T.Gerkmann,M.Krawczyk-Becker,and J.Le Roux,”Phase processing for single-channel speech enhancement:History and recent advances,” IEEE Signal Processing Magazine,volume 32,no.2,pp.55-66,March 2015

[0321] [9] E.Vincent,T.Virtanen,and S.Gannot,Audio Source Separation and Speech Enhancement.Wiley,2018

[0322]

[10] D.Matz,E.Cano,and J.Abesser,”New sonorities for early jazz recordings using sound source separation and automatic mixing tools,” in Proc.of the 16th International Society for Music Information Retrieval Conference.Malaga,Spain:ISMIR,October 2015,pp.749-755

[0323]

[11] S.M.Kuo and D.R.Morgan,”Active noise control:a tutorial review,” Proceedings of the IEEE,volume 87,no.6,pp.943-973,June 1999

[0324]

[12] A.McPherson,R.Jack,and G.Moro,”Action-sound latency:Are our tools fast enough?” in Proceedings of the International Conference on New Interfaces for Musical Expression,July 2016

[0325]

[13] C.Rottondi,C.Chafe,C.Allocchio,and A.Sarti,”An overview on networked music performance technologies,” IEEE Access,volume 4,pp.8823-8843,2016

[0326]

[14] S.Liebich,J.Fabry,P.Jax,and P.Vary,”Signal processing challenges for active noise cancellation headphones,” in Speech Communication;13th ITG-Symposium,October 2018,pp.1-5

[0327]

[15] E.Cano,J.Liebetrau,D.Fitzgerald,and K.Brandenburg,”The dimensions of perceptual quality of sound source separation,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2018,pp.601-605

[0328]

[16] P.M.Delgado and J.Herre,”Objective assessment of spatial audio quality using directional loudness maps,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),May 2019,pp.621-625

[0329]

[17] C.H.Taal,R.C.Hendriks,R.Heusdens,and J.Jensen,”An algorithm for intelligibility prediction of time-frequency weighted noisy speech,” IEEE Transactions on Audio,Speech,and Language Processing,volume 19,no.7,pp.2125-2136,September 2011

[0330]

[18] M.D.Plumbley,C.Kroos,J.P.Bello,G.Richard,D.P.Ellis,and A.Mesaros,Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop(DCASE2018).Tampere University of Technology.Laboratory of Signal Processing,2018

[0331]

[19] R.Serizel,N.Turpault,H.Eghbal-Zadeh,and A.Parag Shah,”Large-Scale Weakly Labeled Semi-Supervised Sound Event Detection in Domestic Environments,” July 2018,submitted to DCASE2018 Workshop

[0332]

[20] L.JiaKai,”Mean teacher convolution system for dcase 2018 task 4,” DCASE2018 Challenge,Tech.Rep.,September 2018

[0333]

[21] G.Parascandolo,H.Huttunen,and T.Virtanen,”Recurrent neural networks for polyphonic sound event detection in real life recordings,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),March 2016,pp.6440-6444

[0334]

[22] E.C,Cakir and T.Virtanen,”End-to-end polyphonic sound event detection using convolutional recurrent neural networks with learned time-frequency representation input,” in Proc.of International Joint Conference on Neural Networks(IJCNN),July 2018,pp.1-7

[0335]

[23] Y.Xu,Q.Kong,W.Wang,and M.D.Plumbley,”Large-Scale Weakly Supervised Audio Classification Using Gated Convolutional Neural Network,” in Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),Calgary,AB,Canada,2018,pp.121-125

[0336]

[24] B.Frenay and M.Verleysen,”Classification in the presence of label noise:A survey,” IEEE Transactions on Neural Networks and Learning Systems,volume 25,no.5,pp.845-869,May 2014

[0337]

[25] E.Fonseca,M.Plakal,D.P.W.Ellis,F.Font,X.Favory,and X.Serra,”Learning sound event classifiers from web audio with noisy labels,” in Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),Brighton,UK,2019

[0338]

[26] M.Dorfer and G.Widmer,”Training general-purpose audio tagging networks with noisy labels and iterative self-verification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop(DCASE2018),Surrey,UK,2018

[0339]

[27] S.Adavanne,A.Politis,J.Nikunen,and T.Virtanen,”Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,” IEEE Journal of Selected Topics in Signal Processing,pp.1-1,2018

[0340]

[28] Y.Jung,Y.Kim,Y.Choi,and H.Kim,”Joint learning using denoising variational autoencoders for voice activity detection,” in Proc.of Interspeech,September 2018,pp.1210-1214

[0341]

[29] F.Eyben,F.Weninger,S.Squartini,and B.Schuller,”Real-life voice activity detection with LSTM recurrent neural networks and an application to hollywood movies,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing,May 2013,pp.483-487

[0342]

[30] R.Zazo-Candil,T.N.Sainath,G.Simko,and C.Parada,”Feature learning with raw-waveform CLDNNs for voice activity detection,” in Proc.of INTERSPEECH,2016

[0343]

[31] M.McLaren,Y.Lei,and L.Ferrer,”Advances in deep neural network approaches to speaker recognition,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2015,pp.4814-4818

[0344]

[32] D.Snyder,D.Garcia-Romero,G.Sell,D.Povey,and S.Khudanpur,”X-vectors:Robust DNN embeddings for speaker recognition,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2018,pp.5329-5333

[0345]

[33] M.McLaren,D.Castan,M.K.Nandwana,L.Ferrer,and E.Yilmaz,”How to train your speaker embeddings extractor,” in Odyssey,2018

[0346]

[34] S.O.Sadjadi,J.W.Pelecanos,and S.Ganapathy,”The IBM speaker recognition system:Recent advances and error analysis,” in Proc.of Interspeech,2016,pp.3633-3637

[0347]

[35] Y.Han,J.Kim,and K.Lee,”Deep convolutional neural networks for predominant instrument recognition in polyphonic music,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 25,no.1,pp.208-221,January 2017

[0348]

[36] V.Lonstanlen and C.-E.Cella,”Deep convolutional networks on the pitch spiral for musical instrument recognition,” in Proceedings of the 17th International Society for Music Information Retrieval Conference.New York,USA:ISMIR,2016,pp.612-618

[0349]

[37] S.Gururani,C.Summers,and A.Lerch,”Instrument activity detection in polyphonic music using deep neural networks,” in Proceedings of the 19th International Society for Music Information Retrieval Conference.Paris,France:ISMIR,September 2018,pp.569-576

[0350]

[38] J.Schlutter and B.Lehner,”Zero mean convolutions for level-invariant singing voice detection,” in Proceedings of the 19th International Society for Music Information Retrieval Conference.Paris,France:ISMIR,September 2018,pp.321-326

[0351]

[39] S.Delikaris-Manias,D.Pavlidi,A.Mouchtaris,and V.Pulkki,”DOA estimation with histogram analysis of spatially constrained active intensity vectors,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),March 2017,pp.526-530

[0352]

[40] S.Chakrabarty and E.A.P.Habets,”Multi-speaker DOA estimation using deep convolutional networks trained with noise signals,” IEEE Journal of Selected Topics in Signal Processing,volume 13,no.1,pp.8-21,March 2019

[0353]

[41] X.Li,L.Girin,R.Horaud,and S.Gannot,”Multiple-speaker localization based on direct-path features and likelihood maximization with spatial sparsity regularization,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 25,no.10,pp.1997-2012,October 2017

[0354]

[42] F.Grondin and F.Michaud,”Lightweight and optimized sound source localization and tracking methods for open and closed microphone array configurations,” Robotics and Autonomous Systems,volume 113,pp.63-80,2019

[0355]

[43] D.Yook,T.Lee,and Y.Cho,”Fast sound source localization using two-level search space clustering,” IEEE Transactions on Cybernetics,volume 46,no.1,pp.20-26,January 2016

[0356]

[44] D.Pavlidi,A.Griffin,M.Puigt,and A.Mouchtaris,”Real-time multiple sound source localization and counting using a circular microphone array,” IEEE Transactions on Audio,Speech,and Language Processing,volume 21,no.10,pp.2193-2206,October 2013

[0357]

[45] P.Vecchiotti,N.Ma,S.Squartini,and G.J.Brown,”End-to-end binaural sound localisation from the raw waveform,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),May 2019,pp.451-455

[0358]

[46] Y.Luo,Z.Chen,and N.Mesgarani,”Speaker-independent speech separation with deep attractor network,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 26,no.4,pp.787-796,April 2018

[0359]

[47] Z.Wang,J.Le Roux,and J.R.Hershey,”Multi-channel deep clustering:Discriminative spectral and spatial embeddings for speaker-independent speech separation,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2018,pp.1-5

[0360]

[48] G.Naithani,T.Barker,G.Parascandolo,L.BramslLw,N.H.Pontoppidan,and T.Virtanen,”Low latency sound source separation using convolutional recurrent neural networks,” in Proc.of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics(WASPAA),October 2017,pp.71-75

[0361]

[49] M.Sunohara,C.Haruta,and N.Ono,”Low-latency real-time blind source separation for hearing aids based on time-domain implementation of online independent vector analysis with truncation of non-causal components,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),March 2017,pp.216-220

[0362]

[50] Y.Luo and N.Mesgarani,”TaSNet:Time-domain audio separation network for real-time,single-channel speech separation,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2018,pp.696-700

[0363]

[51] J.Chua,G.Wang,and W.B.Kleijn,”Convolutive blind source separation with low latency,” in Proc.of IEEE International Workshop on Acoustic Signal Enhancement(IWAENC),September 2016,pp.1-5

[0364]

[52] Z.Rafii,A.Liutkus,F.Stoter,S.I.Mimilakis,D.FitzGerald,and B.Pardo,”An overview of lead and accompaniment separation in music,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 26,no.8,pp.1307-1335,August 2018

[0365]

[53] F.-R.Stoter,A.Liutkus,and N.Ito,”The 2018 signal separation evaluation campaign,” in Latent Variable Analysis and Signal Separation,Y.Deville,S.Gannot,R.Mason,M.D.Plumbley,and D.Ward,Eds.Cham:Springer International Publishing,2018,pp.293-305

[0366]

[54] J.-L.Durrieu,B.David,and G.Richard,”A musically motivated midlevel representation for pitch estimation and musical audio source separation,” Selected Topics in Signal Processing,IEEE Journal of,volume 5,no.6,pp.1180-1191,October 2011

[0367]

[55] S.Uhlich,M.Porcu,F.Giron,M.Enenkl,T.Kemp,N.Takahashi,and Y.Mitsufuji,”Improving music source separation based on deep neural networks through data augmentation and network blending,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),2017

[0368]

[56] P.N.Samarasinghe,W.Zhang,and T.D.Abhayapala,”Recent advances in active noise control inside automobile cabins:Toward quieter cars,” IEEE Signal Processing Magazine,volume 33,no.6,pp.61-73,November 2016

[0369]

[57] S.Papini,R.L.Pinto,E.B.Medeiros,and F.B.Coelho,”Hybrid approach to noise control of industrial exhaust systems,” Applied Acoustics,volume 125,pp.102-112,2017

[0370]

[58] J.Zhang,T.D.Abhayapala,W.Zhang,P.N.Samarasinghe,and S.Jiang,”Active noise control over space:A wave domain approach,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 26,no.4,pp.774-786,April 2018

[0371]

[59] X.Lu,Y.Tsao,S.Matsuda,and C.Hori,”Speech enhancement based on deep denoising autoencoder,” in Proc.of Interspeech,2013

[0372]

[60] Y.Xu,J.Du,L.Dai,and C.Lee,”A regression approach to speech enhancement based on deep neural networks,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 23,no.1,pp.7-19,January 2015

[0373]

[61] S.Pascual,A.Bonafonte,and J.Serra,”SEGAN:speech enhancement generative adversarial network,” in Proc.of Interspeech,August 2017,pp.3642-3646

[0374]

[62] F.Weninger,H.Erdogan,S.Watanabe,E.Vincent,J.Le Roux,J.R.Hershey,and B.Schuller,”Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,” in Latent Variable Analysis and Signal Separation,E.Vincent,A.Yeredor,Z.Koldovsky,and P.Tichavsky,Eds.Cham:Springer International Publishing,2015,pp.91-99

[0375]

[63] H.Wierstorf,D.Ward,R.Mason,E.M.Grais,C.Hummersone,and M.D.Plumbley,”Perceptual evaluation of source separation for remixing music,” in Proc.of Audio Engineering Society Convention 143,October 2017

[0376]

[64] J.Pons,J.Janer,T.Rode,and W.Nogueira,”Remixing music using source separation algorithms to improve the musical experience of cochlear implant users,” The Journal of the Acoustical Society of America,volume 140,no.6,pp.4338-4349,2016

[0377]

[65] Q.Kong,Y.Xu,W.Wang,and M.D.Plumbley,”A joint separation-classification model for sound event detection of weakly labelled data,” in Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),March 2018

[0378]

[66] T.v.Neumann,K.Kinoshita,M.Delcroix,S.Araki,T.Nakatani,and R.Haeb-Umbach,”All-neural online source separation,counting,and diarization for meeting analysis,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),May 2019,pp.91-95

[0379]

[67] S.Gharib,K.Drossos,E.Cakir,D.Serdyuk,and T.Virtanen,”Unsupervised adversarial domain adaptation for acoustic scene classification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop(DCASE),November 2018,pp.138-142

[0380]

[68] A.Mesaros,T.Heittola,and T.Virtanen,”A multi-device dataset for urban acoustic scene classification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop,Surrey,UK,2018

[0381]

[69] J.Abesser,M.Gotze,S.Kuhnlenz,R.Grafe,C.Kuhn,T.Clauss,H.Lukashevich,”A Distributed Sensor Network for Monitoring Noise Level and Noise Sources in Urban Environments,” in Proceedings of the 6th IEEE International Conference on Future Internet of Things and Cloud(FiCloud),Barcelona,Spain,pp.318-324.,2018

[0382]

[70] T.Virtanen,M.D.Plumbley,D.Ellis(Eds.),”Computational Analysis of Sound Scenes and Events,” Springer,2018

[0383]

[71] J.Abesser,S.Ioannis Mimilakis,R.Grafe,H.Lukashevich,”Acoustic scene classification by combining autoencoder-based dimensionality reduction and convolutional neural net-works,” in Proceedings of the 2nd DCASE Workshop on Detection and Classification of Acoustic Scenes and Events,Munich,Germany,2017

[0384]

[72] A.Avni,J.Ahrens,M.Geierc,S.Spors,H.Wierstorf,B.Rafaely,”Spatial perception of sound fields recorded by spherical microphone arrays with varying spatial resolution,” Journal of the Acoustic Society of America,133(5),pp.2711-2721,2013

[0385]

[73] E.Cano,D.FitzGerald,K.Brandenburg,”Evaluation of quality of sound source separation algorithms:Human perception vs quantitative metrics,” in Proceedings of the 24th European Signal Processing Conference(EUSIPCO),pp.1758-1762,2016

[0386]

[74] S.Marchand,”Audio scene transformation using informed source separation,” The Journal of the Acoustical Society of America,140(4),p.3091,2016

[0387]

[75] S.Grollmisch,J.Abesser,J.Liebetrau,H.Lukashevich,”Sounding industry:Challenges and datasets for industrial sound analysis(ISA),” in Proceedings of the 27th European Signal Processing Conference(EUSIPCO)(submitted),A Coruna,Spain,2019

[0388]

[76] J.Abesser,M.Muller,”Fundamental frequency contour classification:A comparison between hand-crafted and CNN-based features,” in Proceedings of the 44th IEEE International Conference on Acoustics,Speech,and Signal Processing(ICASSP),2019

[0389]

[77] J.Abesser,S.Balke,M.Muller,”Improving bass saliency estimation using label propagation and transfer learning,” in Proceedings of the 19th International Society for Music Information Retrieval Conference(ISMIR),Paris,France,pp.306-312,2018

[0390]

[78] C.-R.Nagar,J.Abesser,S.Grollmisch,”Towards CNN-based acoustic modeling of seventh chords for recognition chord recognition,” in Proceedings of the 16th Sound&Music Computing Conference(SMC)(submitted),Malaga,Spain,2019

[0391]

[79] J.S.Gomez,J.Abesser,E.Cano,”Jazz solo instrument classification with convolutional neural networks,source separation,and transfer learning”,in Proceedings of the 19th International Society for Music Information Retrieval Conference(ISMIR),Paris,France,pp.577-584,2018

[0392]

[80] J.R.Hershey,Z.Chen,J.Le Roux,S.Watanabe,”Deep clustering:Discriminative embeddings for segmentation and separation,” in Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),pp.31-35,2016

[0393]

[81] E.Cano,G.Schuller,C.Dittmar,”Pitch-informed solo and accompaniment separation towards its use in music education applications”,EURASIP Journal on Advances in Signal Processing,2014:23,pp.1-19

[0394]

[82] S.I.Mimilakis,K.Drossos,J.F.Santos,G.Schuller,T.Virtanen,Y.Bengio,”Monaural Singing Voice Separation with Skip-Filtering Connections and Recurrent Inference of Time-Frequency Mask,” in Proceedings of the IEEE International Conference on Acoustics,Speech,and Signal Processing(ICASSP),Calgary,Canada,S.721-725,2018

[0395]

[83] J.F.Gemmeke,D.P.W.Ellis,D.Freedman,A.Jansen,W.Lawrence,R.C.Moore,M.Plakal,M.Ritter,”Audio Set:An ontology and human-labeled dataset for audio events,” in Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),New Orleans,USA,2017

[0396]

[84] Kleiner,M.”Acoustics and Audio Technology,”.3rd ed.USA:J.Ross Publishing,2012

[0397]

[85] M.Dickreiter,V.Dittel,W.Hoeg,M.Wohr,M.,,Handbuch der Tonstudiotechnik,” A.medienakademie(Eds).7th edition,Vol.1.,Munich:K.G.Saur Verlag,2008

[0398]

[86] F.Muller,M.Karau.,,Transparant hearing,” in:CHI ,02 Extended Abstracts on Human Factors in Computing Systems(CHI EA ’02),Minneapolis,USA,pp.730-731,April 2002

[0399]

[87] L.Vieira.”Super hearing:a study on virtual prototyping for hearables and hearing aids,” Master Thesis,Aalborg University,2018.Available:https: / / projekter.aau.dk / projekter / files / 287515943 / MasterThesis_Luis.pdf

[0400]

[88] Sennheiser,”AMBEO Smart Headset,” [Online].Available: https: / / de-de.sennheiser.com / finalstop [Accessed:March 1,2019]

[0401]

[89] Orosound ”Tilde Earphones” [Online].Available: https: / / www.orosound.com / tilde-earphones / [Accessed;March 1,2019]

[0402]

[90] Brandenburg, K., Cano Ceron, E., Klein, F., Kollmer, T., Lukashevich, H., Neidhardt, A., Nowak, J., Sloma, U., and Werner, S., “Personalized auditory reality,” in 44th Annual Conference on Acoustics (DAGA), Garching near Munich, German Society for Acoustics (DEGA), 2018

[0403]

[91] US 2015 195641 A1,Application date:January 6,2014;published on July 9,2015.

Claims

1. an analyzer (152) for determining a plurality of binaural room impulse responses; a loudspeaker signal generator (154) for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source; Equipped with the analyzer (152) is configured to determine the plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account an effect resulting from a user wearing headphones. system.

2. The system comprises the headphones, the headphones are configured to output the at least two loudspeaker signals; The system of claim 1 .

3. The headphones include two headphone capsules and at least one microphone for measuring sound in each of the two headphone capsules; the at least one microphone is configured to measure the sound and is disposed in each of the two headphone capsules; the analyzer (152) is configured to perform the determination of the plurality of binaural room impulse responses using the measurements of the at least one microphone in each of the two headphone capsules.

3. A system according to claim 1 or 2.

4. the at least one microphone in each of the two headphone capsules is configured to generate one or more recordings of a sound situation in a reproduction room prior to reproduction of the at least two loudspeaker signals by the headphones, determine an estimate of a first sound signal of at least one sound source from the one or more recordings, and determine a binaural room impulse response of the plurality of binaural room impulse responses for the sound source in the reproduction room. The system of claim 3.

5. the at least one microphone in each of the two headphone capsules is configured to generate, during playback of the at least two loudspeaker signals by the headphones, one or more further recordings of the sound situation in the reproduction room, subtract an augmented signal from these one or more further recordings, determine the estimate of the first sound signal from one or more sound sources, and determine the binaural room impulse response of the plurality of binaural room impulse responses for the sound sources in the reproduction room. The system of claim 4.

6. the analyzer (152) is configured to determine an acoustic room characteristic of the reproduction room and to adapt the plurality of binaural room impulse responses in response to the acoustic room characteristic.

6. A system according to claim 4 or 5.

7. the at least one microphone is disposed in each of the two headphone capsules to measure sounds near the entrance of the ear canal; A system according to any one of claims 4 to 6.

8. the system includes one or more further microphones outside the two headphone capsules for measuring the sound situation in the reproduction room; A system according to any one of claims 4 to 7.

9. the headphones include a headphone band, and at least one of the one or more further microphones is disposed on the headphone band. The system of claim 8.

10. the loudspeaker signal generator (154) is configured to generate the at least two loudspeaker signals by convolving each of the plurality of binaural room impulse responses with a source signal of a plurality of one or more source signals. A system according to any one of claims 1 to 9.

11. the analyzer (152) is configured to determine at least one of the plurality of binaural room impulse responses in response to a movement of the headphones. A system according to any one of claims 1 to 10.

12. The system includes a sensor for determining movement of the headphones. The system of claim 11.

13. 1. A system for assisting selective hearing, comprising: a detector (110) for detecting source signal portions of one or more sound sources using at least two received microphone signals of the auditory environment; a position determiner (120) for assigning position information to each of said one or more sound sources; a sound type classifier (130) for assigning a sound signal type to the sound source signal portions of each of the one or more sound sources; a signal portion modifier (140) for modifying the source signal portion of the at least one sound source of the one or more sound sources depending on the sound signal type of the source signal portion of the at least one sound source to obtain a modified sound signal portion of the at least one sound source; Further equipped with said analyzer (152) and said loudspeaker signal generator (154) together form a signal generator (150); the analyzer (152) of the signal generator (150) is configured to generate the plurality of binaural room impulse responses, the plurality of binaural room impulse responses being a plurality of binaural room impulse responses for each of the one or more sound sources, dependent on the position information of the sound sources and on a user's head orientation; the loudspeaker signal generator (154) of the signal generator (150) is configured to generate the at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to the modified audio signal portion of the at least one sound source. The system of claim 5.

14. the detector (110) is configured to detect the source signal portions of the one or more sound sources by using a deep learning model. The system of claim 13.

15. the location determiner (120) is configured to determine the location information for each of the one or more sound sources in response to captured images or recorded video.

15. A system according to claim 13 or 14.

16. the signal portion modifier (140) is configured to select the at least one sound source whose sound source signal portion is to be modified in response to a previously learned user scenario, and to modify the sound source in response to the previously learned user scenario. A system according to any one of claims 13 to 15.

17. The system includes a remote device (190) including the detector (110), the location determiner (120), the sound type classifier (130), the signal portion modifier (140), and the signal generator (150); the remote device is spatially separated from the headphones; 17. A system according to any one of claims 13 to 16.

18. The system of claim 17, wherein the remote device (190) is a smartphone.

19. 1. A method comprising: determining, by an analyzer (152) of the system, a plurality of binaural room impulse responses; generating at least two loudspeaker signals by a loudspeaker signal generator (154) of the system in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source; Including, the plurality of binaural room impulse responses are determined by the analyzer (152) such that each of the plurality of binaural room impulse responses takes into account an effect due to a user wearing headphones. method.

20. A computer program having a program code for performing the method according to claim 19, when the computer program is executed by a computer or signal processor.

Images