Systems and methods for assisting selective hearing

Abstract

A system and a corresponding method for assisting selective hearing are provided. The system comprises a detector (110) for detecting an audio source signal portion of one or more audio sources by using at least two received microphone signals of an auditory environment. In addition, the system comprises a position determiner (120) for assigning position information to each of the one or more audio sources. In addition, the system comprises an audio type classifier (130) for assigning an audio source signal type to the audio source signal portion of each of the one or more audio sources. In addition, the system comprises a signal portion modifier (140) for changing the audio source signal portion of at least one of the one or more audio sources according to the audio signal type of the audio source signal portion of the at least one audio source so as to obtain a modified audio signal portion of the at least one audio source. In addition, the system comprises a signal generator (150) for generating a plurality of binaural room impulse responses for each of the one or more audio sources according to the position information of the audio source and the orientation of the user's head, and for generating at least two speaker signals according to the plurality of binaural room impulse responses and according to the modified audio signal portion of the at least one audio source. In addition, a device and a corresponding method for determining one or more indoor acoustic parameters are provided. The device is configured to obtain microphone data including one or more microphone signals. In addition, the device is configured to obtain tracking data about the position and / or orientation of the user. In addition, the device is configured to determine one or more acoustic parameters based on the microphone data and based on the tracking data.

Description

Technical Field

[0001] The present invention relates to aspects of spatial recording, analysis, reproduction and perception, and in particular to binaural analysis and synthesis. Background Art

[0002] Selective hearing (SH) refers to the ability of listeners to direct their attention to a certain sound source or to multiple sound sources in their auditory scene. This, in turn, means that listeners pay less attention to sources that are not of interest.

[0003] Thus, human listeners are able to communicate in noisy environments. This usually exploits different aspects: when listening with both ears, there are direction-dependent time and level differences in sound as well as different spectral colorations depending on the direction. This enables hearing to determine the direction of the sound source and focus on it.

[0004] Furthermore, in the case of natural sound sources (especially speech), signal parts of different frequencies are coupled in time. In this way, hearing is able to distinguish different sound sources even when listening with only one ear. In binaural hearing, both aspects are used together. Furthermore, loud interfering sources that can be easily localized can be actively ignored, so to speak.

[0005] In the literature, the concept of selective hearing is related to other terms such as assisted listening [1], virtual and amplified auditory environments [2]. Assisted listening is a broader term that includes virtual, amplified and SH applications.

[0006] According to the prior art, classical hearing devices work in a monophonic manner, i.e. the signal processing for the left and right ear is completely independent with regard to frequency response and dynamic compression. As a result, the time, level and frequency differences between the human ear signals are lost.

[0007] Modern, so-called binaural hearing devices couple the correction factors of the two hearing devices. Usually, they have several microphones, however, usually only the microphone with the "most speech-like" signal selected but without calculating beamforming. In complex hearing situations, the desired and undesired sound signals are amplified in the same way, so that the focus on the desired sound components is not supported.

[0008] In the field of hands-free devices, for example for telephones, several microphones are already used today and a so-called beam is calculated from the individual microphone signals: the sound coming from the direction of the beam is amplified, the sound from other directions is reduced. Today's methods learn constant sounds in the background (e.g. engine noise and wind noise in a car), learn loud interferences that are well localizable by another beam and subtract these interferences from the used signal (e.g. generalized sidelobe canceller). Sometimes, telephone systems use detectors that detect static properties of speech in order to suppress everything that is not structured like speech. In hands-free devices, ultimately only a mono signal is transmitted, and spatial information that would be of interest for the capture situation is lost in the transmission path, specifically providing the illusion that "one person is there", especially in situations where several speakers are talking to each other. By suppressing non-speech signals, important information about the acoustic environment of the conversation partners is lost, which can hinder communication.

[0009] Humans are essentially able to "selectively listen" and consciously focus on individual sound sources in their surroundings. Automatic systems for selective hearing through artificial intelligence (AI) must first learn the underlying concepts. Automatic decomposition of an acoustic scene (scene decomposition) first requires the detection and classification of all active sound sources, and then their separation so that they can be further processed, amplified, or attenuated as separate audio objects.

[0010] The research field of auditory scene analysis attempts to detect and classify temporally localized sound events such as footsteps, claps, or shouts, as well as more global acoustic scenes such as concerts, restaurants, or supermarkets, based on recorded audio signals. In this context, current approaches exclusively use methods from the fields of artificial intelligence (AI) and deep learning. This involves data-driven learning of deep neural networks that learn to detect characteristic patterns in audio signals based on extensive training

[70] . Most importantly, inspired by advances in the research fields of image processing (computer vision) and speech processing (natural language processing), as a general rule, a mixture of convolutional neural networks for two-dimensional pattern detection in spectrogram representations and recurrent layers (recurrent neural networks) for temporal modeling of sounds is used.

[0011] For audio analysis, there are a number of specific challenges to address. Due to their complexity, deep learning models are very data hungry. Compared to the research areas of image processing and speech processing, only relatively small datasets are available for audio processing. The largest dataset is Google's AudioSet dataset

[83] , which has approximately 2 million sound examples and 632 different sound event classifications, with most datasets used in research being significantly smaller. This small amount of training data can be addressed, for example, by transfer learning, where a model pre-trained on a larger dataset is subsequently fine-tuned to a smaller dataset with new categories determined for the use case (fine-tuning)

[77] . In addition, semi-supervised learning methods are used in order to involve the often large amount of available unannotated audio data in the training.

[0012] Another significant difference compared to image processing is that in the case of simultaneously audible acoustic events, the sound objects are not masked (as in the case of images), but rather there is a complex phase-dependent overlap. Current algorithms in deep learning use so-called “attention” mechanisms, which, for example, enable the model to focus on classifying certain time periods or frequency ranges

[23] . The detection of sound events is further complicated by the high variance of their duration. Algorithms should be able to robustly detect both very short events such as pistol shots as well as long events such as a passing train.

[0013] Due to the strong dependence of the models on the acoustic conditions when the training data was recorded, they often exhibit unexpected behavior in new acoustic environments, e.g., they differ in terms of spatial reverberation or the location of the microphones. Different solutions have been developed to mitigate this problem. For example, data augmentation methods try to achieve higher robustness and invariance of the models by simulating different acoustic conditions

[68] and artificial overlap of different sound sources. Furthermore, parameters in complex neural networks can be adjusted in different ways to avoid overtraining and specialization on the training data while achieving better generalization to unseen data. In recent years, different algorithms have been proposed for “domain adaptation”

[67] in order to adapt previously trained models to new application conditions. In the usage scenario planned in this project, inside headphones, the real-time capability of the sound source detection algorithm is of fundamental importance. Here, a trade-off has to be made between the complexity of the neural network and the maximum possible number of computational operations on the underlying computing platform. Even if a sound event has a long duration, it still has to be detected as quickly as possible in order to start the corresponding source separation.

[0014] At Fraunhofer IDMT, a great deal of research work has been carried out in the field of automatic sound source detection in recent years. In the research project “StadtLärm”, a distributed sensor network has been developed that can measure noise levels and classify between 14 different acoustic scenarios and event categories based on audio signals recorded at different locations within a city

[69] . In this case, the processing in the sensors is performed in real time on the embedded platform Raspberry Pi 3. Previous work investigated novel methods for data compression of spectrograms based on autoencoder networks

[71] . Recently, great progress has been made in the field of music signal processing (music information retrieval) by using deep learning methods in applications such as music transcription

[76] ,

[77] , chord detection

[78] and instrument detection

[79] . In the field of industrial audio processing, new datasets have been created and deep learning methods have been used, for example, to monitor the acoustic state of electric motors

[75] .

[0015] The scenario addressed in this embodiment assumes several sound sources whose number and type are initially unknown and whose number and type can change over time. For sound source separation, several sources with similar characteristics (e.g., several speakers) are particularly challenging

[80] .

[0016] To achieve high spatial resolution, several microphones must be used in an array

[72] . Compared to traditional audio recordings in mono (1 channel) or stereo (2 channels), this recording scenario enables precise localization of sound sources around the listener.

[0017] Source separation algorithms often leave artifacts, such as distortion and crosstalk between sources [5], which are generally perceived by listeners as disturbing. Such artifacts can be partially masked and thus reduced by remixing the tracks

[10] .

[0018] To enhance “blind” source separation, additional information is often used, such as the number and type of detected sources or the estimated spatial location of the sources (informed source separation

[74] ). For conferences with several active speakers, current analysis systems can simultaneously estimate the number of speakers, determine their respective temporal activities, and subsequently isolate them via source separation

[66] .

[0019] At Fraunhofer IDMT, a lot of research has been performed in recent years on the perception-based evaluation of sound source separation algorithms

[73] .

[0020] In the field of music signal processing, a real-time capable algorithm has been developed for separating solo instruments from accompaniment instruments using the fundamental frequency estimate of the solo instrument as additional information

[81] . An alternative approach based on deep learning methods for separating vocals from complex musical compositions was proposed in

[82] . Specialized source separation algorithms have also been developed for applications in the context of industrial audio analysis [7].

[0021] Headphones significantly influence the acoustic perception of the surrounding environment. Depending on the design of the headphone, sound incident on the ear is attenuated to varying degrees. In-ear headphones completely block the ear canal

[85] . Closed-back headphones, which surround the pinna, also acoustically strongly cut off the listener from the outside world. Open-back and semi-open-back headphones allow sound to pass through completely or partially

[84] . In many applications in everyday life, headphones are expected to provide a stronger isolation from unwanted ambient sound than their design type would suggest.

[0022] Interfering influences from the outside can also be attenuated by active noise control (ANC). This is achieved by recording the incident sound signals via the microphone of the headphones and then reproducing them via the loudspeakers in such a way that these sound parts and the sound parts that penetrate the headphones cancel each other out by interference. Overall, this allows for a strong acoustic isolation from the surroundings. However, in many everyday situations, this is accompanied by dangers, which is why it is desirable to be able to switch this function on intelligently as required.

[0023] Firstly, products are able to allow microphone signals to pass into the earphones in order to reduce passive isolation. Thus, in addition to prototypes

[86] , products already exist that advertise the function of “transparent listening”. For example, Sennheiser offers this function with its AMBEO headphones

[88] , while Bragi offers it with its product “The Dash Pro”. However, this possibility is only the beginning. In the future, this function will be greatly expanded so that, in addition to completely switching the ambient sound on and off, individual signal parts (e.g. only speech or alarm signals) can be made fully audible when required. The French company Orosound enables the wearer of the “Tilde Earphones”

[89] to adjust the intensity of the ANC using a slider. In addition, the voice of the conversation partner can also be heard during activated ANC. However, this only works if the conversation partners are located face to face in a 60° cone. Direction-independent adjustment is not possible.

[0024] Patent application publication US 2015 195641 A1 (see

[91] ) discloses a method for generating an auditory environment for a user. In this case, the method includes: receiving a signal representing an ambient auditory environment of the user, processing the signal by using a microprocessor to identify at least one sound type of a plurality of sound types in the ambient auditory environment. In addition, the method includes: receiving a user preference for each of the plurality of sound types, modifying the signal for each sound type in the ambient auditory environment, and outputting the modified signal to at least one speaker to generate an auditory environment for the user. Summary of the invention

[0025] A system for assisting selective hearing is provided. The system includes a detector for detecting an audio source signal portion of one or more audio sources by using at least two received microphone signals of an auditory environment. In addition, the system includes a position determiner for assigning position information to each of the one or more audio sources. In addition, the system includes an audio type classifier for assigning an audio signal type to the audio source signal portion of each of the one or more audio sources. In addition, the system includes a signal portion modifier for changing the audio source signal portion of at least one audio source according to the audio signal type of the audio source signal portion of at least one of the one or more audio sources, so as to obtain a modified audio signal portion of at least one audio source. In addition, the system includes a signal generator for generating a plurality of binaural room impulse responses for each of the one or more audio sources according to the position information of the audio source and the orientation of the user's head, and for generating at least two speaker signals according to the plurality of binaural room impulse responses and according to the modified audio signal portion of at least one audio source.

[0026] In addition, a method for assisting selective hearing is provided. The method comprises:

[0027] - Detecting audio source signal portions of one or more audio sources by using at least two received microphone signals of an auditory environment.

[0028] - Assign position information to each of the one or more audio sources.

[0029] - Assign an audio signal type to an audio source signal portion of each of one or more audio sources.

[0030] - changing the audio source signal portion of at least one audio source according to the audio signal type of the audio source signal portion of at least one audio source of the one or more audio sources so as to obtain a modified audio signal portion of the at least one audio source. And:

[0031] - for each of the one or more audio sources, generating a plurality of binaural room impulse responses for each audio source based on position information of the audio source and an orientation of the user's head, and for generating at least two loudspeaker signals based on the plurality of binaural room impulse responses and based on the modified audio signal portion of at least one audio source.

[0032] Furthermore, a computer program having a program code for executing the above method is provided.

[0033] In addition, a device for determining one or more indoor acoustic parameters is provided. The device is configured to obtain microphone data including one or more microphone signals. In addition, the device is configured to obtain tracking data about the position and / or orientation of a user. In addition, the device is configured to determine one or more indoor acoustic parameters based on the microphone data and based on the tracking data.

[0034] In addition, a method for determining one or more room acoustic parameters is provided. The method comprises:

[0035] - obtain microphone data comprising one or more microphone signals.

[0036] - Obtain tracking data about the user's location and / or orientation. And:

[0037] - determining one or more room acoustic parameters based on the microphone data and based on the tracking data.

[0038] Furthermore, a computer program having a program code for executing the above method is provided.

[0039] Among other things, embodiments are based on merging and combining different technologies for assistive hearing in a technical system to enable enhancements in sound quality and quality of life (e.g., desired sounds are louder, desired sounds are quieter, better speech intelligibility) for both people with normal hearing and people with hearing impairment.

[0040] Next, preferred embodiments of the present invention will be described with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In the attached picture:

[0042] Figure 1 A system for assisting selective hearing according to an embodiment is shown.

[0043] Figure 2 A system is shown that also includes a user interface according to an embodiment.

[0044] Figure 3 A system comprising a hearing device with two corresponding loudspeakers according to an embodiment is shown.

[0045] Figure 4 A system including a housing structure and two loudspeakers according to an embodiment is shown.

[0046] Figure 5 A system including a headset with two speakers according to an embodiment is shown.

[0047] Figure 6 A system is shown comprising a remote device 190 comprising a detector and a position determiner and an audio type classifier and a signal portion modifier and a signal generator according to an embodiment.

[0048] Figure 7 A system including five subsystems according to an embodiment is shown.

[0049] Figure 8 A corresponding scenario according to an embodiment is shown.

[0050] Fig. 9 A scene with four external sound sources according to an embodiment is shown.

[0051] Fig.10 A processing workflow of a SH application according to an embodiment is shown. DETAILED DESCRIPTION

[0052] In modern life, glasses help many people to better perceive their environment. For hearing, there are hearing aids, however, even people with normal hearing can be helped by smart systems in many cases:

[0053] Often, the surrounding area is too noisy, only certain sounds are disturbing, and people want to listen selectively. The human brain is already good at this, but further intelligent assistance in the future could significantly improve this selective hearing. In order to realize such "intelligent hearables" (hearing devices or hearing aids), technical systems must analyze the (acoustic) environment and identify individual sound sources in order to be able to process them separately. There is research work on this topic, however, the analysis and processing of the entire acoustic environment in real time (transparent to our ears) and with high sound quality (so that what is heard is not indistinguishable from the normal acoustic environment) has not yet been achieved with conventional technology.

[0054] Improved concepts for machine listening are provided below.

[0055] Figure 1 A system for assisting selective hearing according to an embodiment is shown.

[0056] The system comprises a detector 110 for detecting audio source signal portions of one or more audio sources by using at least two received microphone signals of an auditory environment (or listening environment).

[0057] Furthermore, the system comprises a position determiner 120 for assigning position information to each of the one or more audio sources.

[0058] Furthermore, the system comprises an audio type classifier 130 for assigning an audio signal type to the audio source signal portion of each of the one or more audio sources.

[0059] Furthermore, the system comprises a signal portion modifier 140 for modifying an audio source signal portion of at least one audio source of the one or more audio sources depending on its audio signal type so as to obtain a modified audio signal portion of the at least one audio source.

[0060] Furthermore, the system comprises a signal generator 150 for generating, for each of the one or more audio sources, a plurality of binaural room impulse responses for each audio source according to position information of the audio source and an orientation of a user's head, and for generating at least two speaker signals according to the plurality of binaural room impulse responses and according to the modified audio signal portion of at least one audio source.

[0061] According to an embodiment, for example, the detector 110 may be configured to detect audio source signal portions of one or more audio sources by using a deep learning model.

[0062] In an embodiment, for example, the position determiner 120 may be configured to determine, for each of the one or more audio sources, position information based on a captured image or a recorded video.

[0063] According to an embodiment, for example, the position determiner 120 can be configured to: for each of the one or more audio sources, determine the position information according to the video in the following manner: detect the lip movement of a person in the video, and assign the lip movement to an audio source signal portion of one of the one or more audio sources according to the lip movement.

[0064] In an embodiment, for example, the detector 110 may be configured to determine one or more acoustic properties of the auditory environment based on at least two received microphone signals.

[0065] According to an embodiment, for example, the signal generator 150 may be configured to determine a plurality of binaural room impulse responses based on one or more acoustic properties of the auditory environment.

[0066] In an embodiment, for example, the signal portion modifier 140 may be configured to select at least one audio source whose audio source signal portion is modified according to a previously learned user scenario, and to modify the at least one audio source according to the previously learned user scenario.

[0067] According to an embodiment, for example, the system may include a user interface 160 for selecting a previously learned user scenario from a group of two or more previously learned user scenarios. Figure 2 Such a system is shown further comprising such a user interface 160 according to an embodiment.

[0068] In an embodiment, for example, the detector 110 and / or the position determiner 120 and / or the audio type classifier 130 and / or the signal modifier 140 and / or the signal generator 150 may be configured to perform parallel signal processing using a Hough transform or employing multiple VLSI chips or by employing multiple memristors.

[0069] According to an embodiment, for example, the system may comprise a hearing device 170 used as a hearing aid for a hearing-limited and / or hearing-impaired user, wherein the hearing device comprises at least two loudspeakers 171 , 172 for outputting at least two loudspeaker signals. Figure 3 Such a system is shown according to an embodiment comprising such a hearing device 170 with two corresponding loudspeakers 171 , 172 .

[0070] In an embodiment, for example, the system may include at least two speakers 181, 182 for outputting at least two speaker signals, and a shell structure 183 accommodating the at least two speakers, wherein at least one shell structure 183 is suitable for being fixed to a user's head 185 or any other body part of the user. Figure 4 A corresponding system comprising such a housing structure 183 and two loudspeakers 181 , 182 is shown.

[0071] According to an embodiment, for example, the system may include a headset 180 including at least two speakers 181 , 182 for outputting at least two speaker signals. Figure 5 A corresponding earphone 180 with two speakers 181 , 182 is shown according to an embodiment.

[0072] In an embodiment, for example, the detector 110 and the position determiner 120 and the audio type classifier 130 and the signal portion modifier 140 and the signal generator 150 may be integrated into the earphone 180 .

[0073] According to an embodiment, Figure 6As shown, for example, the system may include a remote device 190 including a detector 110 and a position determiner 120 and an audio type classifier 130 and a signal portion modifier 140 and a signal generator 150. In this case, for example, the remote device 190 may be spatially separated from the headset 180.

[0074] In an embodiment, for example, remote device 190 may be a smartphone.

[0075] Among other things, embodiments do not necessarily use a microprocessor, but rather use parallel signal processing steps such as Hough transforms, VLSI chips, or memristors for achieving energy savings, and also for artificial neural networks.

[0076] In an embodiment, the auditory environment is captured and reproduced spatially, which on the one hand uses more than one signal to represent the input signal and on the other hand also uses spatial reproduction.

[0077] In an embodiment, the signal separation is performed by a deep learning (DL) model (e.g., CNN, RCNN, LSTM, Siamese network) and information from at least two microphone channels is processed simultaneously, wherein at least one microphone is present in each hearable device. According to the invention, several output signals (according to the individual sound sources) and their respective spatial positions are determined together by mutual analysis. If the recording device (microphone) is attached to the head, the position of the object changes with the movement of the head. This makes it possible to naturally focus on important / unimportant sounds, for example by turning towards the sound object.

[0078] In some embodiments, for example, the algorithm for signal analysis is based on a deep learning architecture. Alternatively, this uses a variant with an analyzer or a variant with a separation network for localization, detection, and sound separation aspects. An alternative use of generalized cross-correlation (correlation with time offset) adapts the frequency-dependent shadowing / isolation of the head and improves localization, detection, and source separation.

[0079] According to an embodiment, the detector learns different source categories (e.g., speech, vehicle, male / female / child voice, warning sound, etc.) during the training phase. Here, the source separation network is also trained for high signal quality, as well as the localization network with target stimuli to achieve high-precision localization.

[0080] For example, the training steps described above use multi-channel audio data, where the first round of training is usually performed in a laboratory using simulated or recorded audio data. This is followed by training in different natural environments (e.g., living rooms, classrooms, train stations, (industrial) production environments, etc.), i.e., performing transfer learning and domain adaptation.

[0081] Alternatively or additionally, the position detector may be coupled to one or more cameras in order to also determine the visual position of the sound source.For speech, the lip movements and the audio signal from the source separator are correlated, enabling a more precise localization.

[0082] After training, there exists a DL model with a network architecture and associated parameters.

[0083] In some embodiments, the sonification is performed by binaural synthesis. Binaural synthesis provides another advantage: undesired components can not be completely deleted, but they can be reduced to a level that is perceptible but not intrusive. This has another advantage: unexpected additional sources (warning signals, shouting, etc.) that would be missed in the case of complete closure can be perceived.

[0084] According to some embodiments, the analysis of the auditory environment is not only used to separate objects, but also to analyze acoustic properties (e.g., reverberation time, initial time gap). These properties are then employed in binaural synthesis in order to adapt the pre-stored (and possibly personalized) binaural room impulse response (BRIR) to the actual room (or space). By reducing the room divergence, the listener significantly reduces the listening effort when understanding the optimized signal. Minimizing the room divergence has an impact on the externalization of auditory events and therefore on the plausibility of the spatial audio reproduction in the monitoring room. There is no known solution in conventional technology for speech understanding or for general understanding of the optimized signal.

[0085] In an embodiment, the user interface is used to determine which sound sources are selected. According to the present invention, this is done by pre-learning different user scenarios, such as "amplify voice from directly in front" (a conversation with one person), "amplify voice within a range of +-60 degrees" (a group conversation), "suppress music and amplify music" (I don't want to hear the sound of the concert audience), "make everything quiet" (I want to be alone), "suppress all shouting and warning sounds", etc.

[0086] Some embodiments are independent of the hardware used, i.e. both open and closed headphones can be used. The signal processing can be integrated into the headphones, can be in an external device, or can be integrated into the smartphone. Optionally, in addition to reproducing the acoustically recorded and processed signal, the signal can also be reproduced directly from the smartphone (e.g., music, phone calls).

[0087] In other embodiments, an ecosystem of "selective hearing with AI assistance" is provided. The embodiment refers to "Personalized Auditory Reality" (PARty). In this personalized environment, the listener is able to amplify, reduce or modify the defined acoustic objects. In order to create a sound experience adapted to individual needs, a series of analysis and synthesis processes are performed. The research work in the target conversion phase forms an essential ingredient for this.

[0088] Some embodiments enable analysis of the real sound environment and detection of individual acoustic objects, separation, tracking and editability of available objects, and reconstruction and reproduction of modified acoustic scenes.

[0089] In an embodiment, detection of sound events, separation of sound events, and suppression of some sound events are achieved.

[0090] In an embodiment, AI methods (particularly deep learning based methods) are used.

[0091] Embodiments of the present invention contribute to the development of techniques for recording, signal processing and reproduction of spatial audio.

[0092] For example, embodiments create spatiality and three-dimensionality in a multimedia system with interacting users.

[0093] In this case, the embodiments are based on research knowledge of the perceptual and cognitive processes of spatial hearing / listening.

[0094] Some embodiments use two or more of the following concepts:

[0095] Scene decomposition: This includes spatial acoustic detection and parameter estimation of real environments and / or location-dependent sound field analysis.

[0096] Scene representation: This includes representation and recognition of objects and / or environments, and / or efficient representation and storage.

[0097] Scene composition and reproduction: This includes the adaptation and variation of objects and environments, and / or rendering and sonification.

[0098] Quality Assessment: This includes technical and / or auditory quality measurements.

[0099] Microphone positioning: This includes the application of microphone arrays and appropriate audio signal processing.

[0100] Signal conditioning: This includes feature extraction as well as dataset generation for ML (machine learning).

[0101] Estimation of room and ambient acoustics: This includes on-site measurement and estimation of room acoustic parameters and / or providing room acoustic characterization for source separation and ML.

[0102] Auralization: This includes spatial audio reproduction with auditory adaptation to the environment and / or validation and evaluation and / or functional proof and quality estimation.

[0103] Figure 8 A corresponding scenario according to an embodiment is shown.

[0104] The embodiments combine concepts for detection, classification, separation, localization, and enhancement of sound sources, highlighting the latest advances in each area and indicating the connections between them.

[0105] A coherent concept is provided below that is able to combine / detect / classify / localize and separate / enhance sound sources in order to provide the flexibility and robustness required for real-life SH.

[0106] Furthermore, embodiments provide low latency concepts suitable for real-time performance when handling the dynamics of real-life auditory scenes.

[0107] Some embodiments use concepts for deep learning, machine listening, and smart headphones (smart hearables) to enable listeners to selectively modify their auditory scene.

[0108] Embodiments provide a listener with the possibility to selectively enhance, attenuate, suppress or modify sound sources in an auditory scene through a hearing device such as headphones, earphones or the like.

[0109] Fig. 9 A scene with four external sound sources according to an embodiment is shown.

[0110] exist Fig. 9 In the example above, the user is the center of the auditory scene. In this case, four external sound sources (S1 to S4) are active around the user. The user interface enables the listener to influence the auditory scene. Sources S1 to S4 can be attenuated, boosted or suppressed via their corresponding sliders. Figure 1 It can be seen that the listener can define sound sources or sound events that should be retained in the auditory scene or suppressed from the auditory scene. Figure 1 In the example, the background noise of the city should be suppressed, while the alarm or phone ringing should be retained. At any time, the user has the possibility to reproduce (or play) additional audio streams such as music or radio via the hearing device.

[0111] The user is usually the center of the system and controls the auditory scene via the control unit. The user can use e.g. Fig. 9 The auditory scene may be modified using the user interface shown or using any type of interaction such as voice control, gestures, gaze direction, etc. Once the user has provided feedback to the system, the next step consists of the detection / classification / localization phase. In some cases, such as if the user wishes to keep any speech occurring in the auditory scene, only detection may be required. In other cases, such as if the user wishes to keep fire alarms in the auditory scene but not ringing phones or office noises, classification may be required. In some cases, only the location of the source is relevant to the system. For example, Fig. 9 This is the case with the four sources in : the user can decide to remove or attenuate a sound source coming from a certain direction, regardless of the type or characteristics of the source.

[0112] Fig.10 A processing workflow of a SH application according to an embodiment is shown.

[0113] First, in Fig.10 The auditory scene is modified during the separation / enhancement stage of the sound source. This can be achieved by suppressing, attenuating or enhancing a sound source (e.g., or some sound sources). Fig.10 As shown, an additional processing candidate in SH is noise control, which aims to remove or minimize background noise in the auditory scene. Perhaps the most popular and widespread noise control technique is active noise control (ANC)

[11] .

[0114] Selective hearing differs from virtually enhanced auditory environments by limiting it to those applications in which only real audio sources are modified in the auditory scene, without attempting to add any virtual sources to the scene.

[0115] From a machine listening perspective, selective hearing applications require techniques to automatically detect, localize, classify, separate, and enhance sound sources. To further clarify the terminology regarding selective hearing, we define the following terms, emphasizing their differences and relationships.

[0116] In embodiments, for example, source localization refers to the ability to detect the location of a sound source in an auditory scene. In the context of audio processing, source location typically refers to the direction of arrival (DOA) of a given source, which can be given in 2D coordinates (azimuth) or 3D coordinates when it includes elevation. Some systems also estimate the distance from the source to the microphone as position information [3]. In the context of music processing, location typically refers to the translation of the source in the final mix, and is usually given as an angle in degrees [4].

[0117] According to an embodiment, for example, using sound source detection refers to the ability to determine whether any instances of a given sound source type are present in an auditory scene. An example of a detection task is determining whether there are any speakers in the scene. In this context, determining the number of speakers in the scene or the identities of the speakers is beyond the scope of sound source detection. Detection can be understood as a binary classification task, where the categories correspond to "source present" and "source not present".

[0118] In an embodiment, for example, using sound source classification, a class label from a set of predefined categories is assigned to a given sound source or a given sound event. An example of a classification task is to determine whether a given sound source corresponds to speech, music, or ambient noise. Sound source classification and detection are closely related concepts. In some cases, the classification system includes a detection phase that considers "no class" as one of the possible labels. In these cases, the system implicitly learns to detect the presence or absence of a sound source and does not enforce the assignment of a class label when there is not enough evidence that any source is active.

[0119] According to an embodiment, source separation is used, for example, to extract a given sound source from an audio mixture or auditory scene. An example of source separation is extracting a vocal from an audio mixture where, in addition to the singer, other instruments are also playing simultaneously [5]. Source separation becomes relevant in selective auditory scenarios as it allows suppressing sound sources that are not of interest to the listener. Some sound separation systems implicitly perform a detection task before extracting the sound source from the mix. However, this is not necessarily a rule, and therefore, we emphasize the distinction between these tasks. Furthermore, separation is often used as a preprocessing stage for other types of analysis such as source enhancement [6] or classification [7].

[0120] In an embodiment, for example, sound source recognition is used, which goes a step further and aims to identify specific instances of sound sources in an audio signal. Speaker recognition is probably the most common use of source recognition today. The goal of this task is to identify whether a specific speaker is present in a scene. Figure 1 In the example of , the user has selected "speaker X" as one of the sources to be retained in the auditory scene. This requires techniques beyond speech detection and classification and requires speaker-specific models that allow such accurate identification.

[0121] Depending on the embodiment, for example, source enhancement is used to refer to the process of increasing the prominence of a given sound source in an auditory scene [8]. In the case of speech signals, the goal is often to improve their perceived quality and intelligibility. A common scenario for speech enhancement is to denoise speech that has been corrupted by noise [9]. In the context of music processing, source enhancement is related to the concept of remixing and is often performed to make one instrument (sound source) more prominent in a mix. Remixing applications often use a sound separation front-end to gain access to the individual sound sources and change the characteristics of the mix

[10] . Although source enhancement can be preceded by a source separation stage, this is not always the case, so we also emphasize the distinction between these terms.

[0122] In the field of sound source detection, classification and identification, for example, some embodiments use one of the following concepts, such as detection and classification of acoustic scenes and events

[18] . In this context, methods have been proposed for audio event detection (AED) in home environments, where the goal is to detect the temporal boundaries of a given sound event within a 10 second recording

[19] ,

[20] . In this particular case, 10 sound event categories were considered, including cat, dog, speech, alarm and running water. Methods for multi-sound sound event (several simultaneous events) detection have also been proposed in the literature

[21] ,

[22] . In

[21] , a method for multi-sound sound event detection is proposed, where a binary activity detector based on a bidirectional long short-term memory (BLSTM) recurrent neural network (RNN) is used to detect a total of 61 sound events from real-life backgrounds.

[0123] Some embodiments, for example, incorporate temporal attention mechanisms to focus on certain regions of the signal for classification in order to handle weakly labeled data

[23] . The problem of noisy labels in classification is particularly relevant to selective hearing applications, where class labels can be very diverse, making high-quality annotations very expensive

[24] . Noisy labels in the sound event classification task are addressed in

[25] , where a noise-resistant loss function based on categorical cross entropy is proposed, as well as methods for evaluating both noisy and manually labeled data. Similarly,

[26] proposes a convolutional neural network (CNN)-based audio event classification system that incorporates a verification step of the noisy labels based on the consensus of the CNN's predictions on multiple segments of training examples.

[0124] For example, some embodiments implement simultaneous detection and localization of sound events. Thus, some embodiments perform detection as a multi-label classification task, such as in

[27] , and the location is given as the 3D coordinates of the direction of arrival (DOA) of each sound event.

[0125] Some embodiments use concepts from SH for voice activity detection and speaker recognition / identification. Voice activity detection has been addressed in noisy environments using denoising autoencoders

[28] , recurrent neural networks

[29] , or as end-to-end systems using raw waveforms

[30] . For speaker recognition applications, a large number of systems have been proposed in the literature

[31] , most of which focus on improving robustness to different conditions, for example through data augmentation or through improved embeddings that facilitate recognition

[32] ~

[34] . Therefore, some embodiments use these concepts.

[0126] Other embodiments use the concept of instrument classification for sound event detection. Instrument classification in both monophonic and polyphonic settings has been addressed in

[35] ,

[36] . In

[35] , the instruments that dominate in a 3 second audio clip are classified into 11 instrument categories, and several aggregation techniques are proposed. Similarly,

[37] proposes an instrument activity detection method that is able to detect instruments with a finer time resolution of 1 second. A lot of research has been done in the field of singing analysis. Specifically, methods such as

[38] have been proposed for detecting segments in audio recordings where the singing is active. Some embodiments use these concepts.

[0127] Some embodiments use one of the concepts discussed below for sound source localization. Sound source localization is closely related to the problem of source counting, since the number of sound sources in an auditory scene is often unknown in real-life applications. Some systems work under the assumption that the number of sources in the scene is known. This is, for example, the case of the model proposed in

[39] , which uses a histogram of activity intensity vectors to localize sources. From a supervised perspective,

[40] proposed a CNN-based algorithm to estimate the DOA of multiple speakers in an auditory scene using phase maps as input representation. In contrast, several works in the literature jointly estimate the number of sources in the scene and their location information. This is the case in

[41] , where a system for multi-talker localization in noisy and reverberant environments is proposed. The system uses a complex-valued Gaussian mixture model (GMM) to estimate the number of sources and their location. The concepts described therein are used by some embodiments.

[0128] Sound source localization algorithms are computationally demanding as they typically involve scanning a large space around the auditory scene

[42] . To reduce the computational requirements in localization algorithms, some embodiments use concepts such as reducing the search space by using clustering algorithms

[43] or by performing multi-resolution searches on well-established methods such as the Steering Response Power Phase Transform (SRP-PHAT) based method

[42] . Other methods impose sparsity constraints and assume that only one sound source is dominant in a given time-frequency domain

[44] . Recently, an end-to-end system for azimuth detection directly from raw waveforms has been proposed in

[45] . Some embodiments use these concepts.

[0129] Some embodiments use concepts described subsequently for sound source separation (SSS), in particular concepts from the fields of speech separation and music separation.

[0130] Specifically, some embodiments use the concept of speaker-independent separation. The separation is performed without any prior information about the speakers in the scene

[46] . Some embodiments also evaluate the spatial location of the speakers in order to perform the separation

[47] .

[0131] Given the importance of computational performance in selective hearing applications, research with the specific goal of achieving low latency is of particular relevance. Some work has been proposed to perform low-latency speech separation (<10ms), where little training data is available

[48] . To avoid the delay caused by framing analysis in the frequency domain, some systems solve the separation problem by carefully designing filters to be applied in the time domain

[49] . Other systems achieve low-latency separation by directly modeling the time domain signal using an encoder-decoder framework

[50] . Conversely, some systems have attempted to reduce the framing delay in frequency domain separation approaches

[51] . These concepts are adopted by some embodiments.

[0132] Some embodiments use the concept of musical sound separation (MSS), such as that for lead instrument-accompaniment separation

[52] , to extract music sources from an audio mixture [5]. These algorithms take the most prominent sound source in the mix, regardless of its class label, and attempt to separate it from the rest of the accompaniment. Some embodiments use the concept for vocal separation

[53] . In most cases, specific source models

[54] or data-driven models

[55] are used to capture the characteristics of the vocals. Although systems such as those proposed in

[55] do not explicitly incorporate a classification or detection stage to achieve separation, the data-driven nature of these methods allows these systems to implicitly learn to detect the vocals with a certain accuracy before separation. Another class of algorithms in the music domain attempts to perform separation using only the location of the sources, without attempting to classify or detect the sources before separation [4].

[0133] Some embodiments use active noise control (ANC) concepts, such as active noise cancellation (ANC). ANC systems are primarily aimed at removing background noise from the headphone user by introducing an anti-noise signal to cancel the background noise

[11] . ANC can be considered a special case of SH and faces the same stringent performance requirements

[14] . Some work has focused on active noise control in specific environments such as car cabins

[56] or industrial scenarios

[57] . The work in

[56] analyzes the cancellation of different types of noise such as road noise and engine noise and requires a unified noise control system that can handle different types of noise. Some work has focused on developing ANC systems to cancel noise over specific spatial regions. In

[58] , spherical harmonics are used as basis functions to represent the noise field to solve ANC in the spatial domain. Some embodiments use the concepts described herein.

[0134] Some embodiments use the concept of sound source enhancement.

[0135] In the context of speech enhancement, one of the most common applications is to enhance speech that has been corrupted by noise. A large body of work has focused on phase processing for monophonic speech enhancement [8]. From a deep neural network perspective, the speech denoising problem has been addressed in

[59] via a denoising autoencoder as a nonlinear regression problem between clean and noisy speech using deep neural networks (DNNs) in

[60] , and in an end-to-end system using generative adversarial networks (GANs) in

[61] . In many cases, speech enhancement is applied as a front-end to an automatic speech recognition (ASR) system, such as in

[62] where speech enhancement is performed via an LSTM RNN. Speech enhancement is also often combined with source separation methods, where the idea is to first extract the speech and then apply enhancement techniques to the isolated speech signal [6]. The concepts described herein are used by some embodiments.

[0136] In most cases, source enhancement related to music refers to applications for creating music remixes. In contrast to speech enhancement, which usually assumes that the speech is only corrupted by noise sources, music applications mostly assume that other sound sources (instruments) are playing simultaneously with the source to be enhanced. For this reason, music remix applications are always provided so that there is a source separation stage before them. In

[10] , for example, early jazz recordings were remixed by applying lead accompaniment and harmonic percussion separation techniques in order to achieve a better sound balance in the mix. Similarly,

[63] studied the use of different vocal separation algorithms in order to change the relative loudness of the vocals and the background tracks, showing that an increase of 6dB could be achieved by introducing a slight but audible distortion in the final mix. In

[64] , the authors studied a method for enhancing the music perception of cochlear implant users by applying sound source separation techniques to achieve a new mix. The concepts described therein are used by some embodiments.

[0137] One of the biggest challenges of selective hearing applications is the stringent requirements on processing time. The complete processing workflow needs to be executed with minimal latency in order to maintain naturalness and perceived quality for the user. The maximum acceptable latency of the system is highly application-dependent and depends on the complexity of the auditory scene. For example, MacPherson et al. proposed 10 milliseconds as an acceptable latency reference for interactive music interfaces

[12] . For music performance on the Internet, the authors in

[13] reported that latency becomes perceptible between 20 milliseconds to 25 milliseconds and 50 milliseconds to 60 milliseconds. However, active noise control / cancellation (ANC) techniques require ultra-low latency processing to achieve better performance. In these systems, the amount of acceptable latency depends on both frequency and attenuation, but can be as low as 1 millisecond for about 5 dB attenuation at frequencies below 200 Hz

[14] . The final consideration in SH applications is the perceived quality of the modified auditory scene. A large amount of work has been devoted to methods for reliably assessing audio quality in different applications

[15] ,

[16] ,

[17] . However, the challenge facing SH is to manage a clear trade-off between processing complexity and perceived quality. Some embodiments use the concepts described therein.

[0138] Some embodiments use the following concepts: for counting / counting and localization as described in

[41] , for localization and detection as described in

[27] , for separation and classification as described in

[65] , and for separation and counting as described in

[66] .

[0139] Some embodiments use concepts for enhancing the robustness of current machine listening methods, as described in

[25] ,

[26] ,

[32] ,

[34] , with emerging directions including domain adaptation

[67] and training on datasets recorded using multiple devices

[68] .

[0140] Some embodiments use concepts such as those described in

[48] for improving the computational efficiency of machine listening methods, or concepts such as those described in

[30] ,

[45] ,

[50] ,

[61] for being able to process raw waveforms.

[0141] Some embodiments implement a unified optimization scheme for detecting / classifying / localizing and separating / enhancing in a combined manner to be able to selectively modify sound sources in a scene, where independent detection, separation, localization, classification, and enhancement methods are reliable and provide the robustness and flexibility required for SH.

[0142] Some embodiments are suitable for real-time processing, where there is a good trade-off between algorithm complexity and performance.

[0143] Some embodiments combine ANC and machine listening. For example, the auditory scene is first classified and then ANC is selectively applied.

[0144] Additional examples are provided below.

[0145] In order to enhance a real auditory environment with virtual audio objects, the conversion function from each of the positions of the audio objects to each of the positions of the listeners in the room must be fully understood.

[0146] The transfer function maps the properties of the sound sources, the direct sound between objects and the user, and all reflections occurring in the room. To ensure a correct spatial audio reproduction of the room acoustics of the real room the listener is currently in, the transfer function must also map the room acoustic properties of the listener's room with sufficient accuracy.

[0147] In an audio system adapted to represent individual audio objects at different locations in a room, the challenge is to properly detect and separate the individual audio objects when there are a large number of audio objects. Furthermore, the audio signals of the objects overlap at the recording or listening position in the room. When the objects and / or the listening position in the room change, the room acoustics and the overlap of the audio signals change.

[0148] The estimation of the room acoustic parameters must be performed fast enough for relative motion. Here, low latency of the estimation is more important than high accuracy. High accuracy is required if the positions of the source and receiver do not change (static case). In the proposed system, the room acoustic parameters as well as the room geometry and the listener position are estimated or extracted from the audio signal stream. The audio signal is recorded in a real environment where the source and receiver can move in any direction and where the source and / or the receiver can change their orientation arbitrarily.

[0149] The audio signal streams can be the result of any microphone arrangement including one or more microphones. These streams are fed to the signal processing stage for pre-processing and / or further analysis. The output is then fed to the feature extraction stage. This stage estimates the room acoustic parameters such as T60 (reverberation time), DRR (direct-reverberation ratio), etc.

[0150] The second data stream is generated by a 6DoF sensor ("six degrees of freedom": three dimensions, one each for position in the room and viewing direction) that captures the orientation and position of the microphone arrangement. The position data stream is fed into the 6DoF signal processing stage for pre-processing or further analysis.

[0151] The output of the 6DoF signal processing, audio feature extraction phase, and the preprocessed microphone streams are fed into a machine learning block where the auditory space or listening room (size, geometry, reflective surfaces) and the positions of the microphone fields in the room are estimated. In addition, a user behavior model is applied in order to achieve a more robust estimate. The model takes into account the limits of human motion (e.g., continuous motion, speed, etc.), as well as the probability distribution of different types of motion.

[0152] Some of the embodiments enable blind estimation of room acoustic parameters by using any microphone arrangement and by adding the user's position and posture information and by analyzing the data using machine learning methods.

[0153] For example, the system according to the embodiment can be used for acoustic augmented reality (AAR). In this case, a virtual room impulse response must be synthesized based on the estimated parameters.

[0154] Some embodiments include removing reverberation from a recorded signal. An example of such an embodiment is a hearing aid for normal hearing people and hearing impaired people. In this case, reverberation can be removed from the input signal of the microphone arrangement with the aid of the estimated parameters.

[0155] Another application is the spatial synthesis of audio scenes generated in rooms other than the current listening space. To this end, room acoustic parameters that are part of the audio scene are adjusted relative to the room acoustic parameters of the listening space.

[0156] In the case of binaural synthesis, the available BRIRs are adapted to the different acoustic parameters of the auditory space for this purpose.

[0157] In an embodiment, an apparatus for determining one or more room acoustic parameters is provided.

[0158] The apparatus is configured to obtain microphone data comprising one or more microphone signals.

[0159] Additionally, the device is configured to obtain tracking data regarding the user's position and / or orientation.

[0160] Furthermore, the apparatus is configured to determine one or more room acoustic parameters from the microphone data and from the tracking data.

[0161] According to an embodiment, for example, the apparatus may be configured to employ machine learning to determine one or more room acoustic parameters from the microphone data and from the tracking data.

[0162] In embodiments, for example, the apparatus may be configured to employ machine learning, as the apparatus may be configured to employ a neural network.

[0163] According to an embodiment, for example, the device may be configured to employ cloud-based processing for machine learning.

[0164] In an embodiment, for example, the one or more room acoustic parameters may include reverberation time.

[0165] According to an embodiment, the one or more room acoustic parameters may include, for example, a direct-reverberation ratio.

[0166] In an embodiment, for example, the tracking data may include x-coordinates, y-coordinates, and z-coordinates marking the location of the user.

[0167] Depending on the embodiment, for example, the tracking data may include pitch, yaw, and roll coordinates to mark the orientation of the user.

[0168] In an embodiment, for example, the apparatus may be configured to transform one or more microphone signals from a time domain to a frequency domain, for example, wherein the apparatus is configured to extract one or more features of the one or more microphone signals in the frequency domain, for example, and wherein the apparatus may be configured to determine one or more indoor acoustic parameters based on the one or more features.

[0169] According to an embodiment, for example, the apparatus may be configured to employ cloud-based processing to extract one or more features.

[0170] In an embodiment, for example, the device may comprise a microphone arrangement of several microphones to record several microphone signals.

[0171] According to an embodiment, for example, the microphone arrangement may be configured to be worn on the body of a user.

[0172] In an embodiment, for example, Figure 1 The above system may further comprise the above device for determining one or more indoor acoustic parameters.

[0173] According to an embodiment, for example, the signal portion modifier 140 may be configured to perform a change in the audio source signal portion of at least one of the one or more audio sources based on at least one of the one or more room acoustic parameters; and / or wherein the signal generator 150 may be configured to perform at least one of generating a plurality of binaural room impulse responses for each of the one or more audio sources based on at least one of the one or more room acoustic parameters.

[0174] Figure 7 A system including five subsystems (Subsystems 1 to 5) according to an embodiment is shown.

[0175] The subsystem 1 comprises a microphone arrangement of one, two or several individual microphones, which may be combined into a microphone field if more than one microphone is available. The positioning and relative arrangement of the microphone / microphones with respect to each other may be arbitrary. The microphone arrangement may be part of a device worn by the user, or it may be a separate device located in the room of interest.

[0176] In addition, subsystem 1 includes a tracking device for obtaining tracking data about the position and / or orientation of the user. For example, the tracking data about the position and / or orientation of the user can be used to determine the translational position of the user in the room and the head posture of the user. Up to 6DoF (six degrees of freedom, such as x-coordinate, y-coordinate, z-coordinate, pitch angle, yaw angle, roll angle) can be measured.

[0177] In this case, for example, the tracking device can be configured to measure tracking data. The tracking device can be positioned on the user's head, or it can be divided into several sub-devices to measure the required degrees of freedom, and it can be placed on the user or not.

[0178] Therefore, the subsystem 1 represents an input interface including a microphone signal input interface 101 and a position information input interface 102 .

[0179] Subsystem 2 includes signal processing of the recorded microphone signals. This includes frequency transformation and / or time domain based processing. In addition, this includes methods for combining different microphone signals to achieve field processing. Feedback from system 4 is possible in order to adjust the parameters of the signal processing in subsystem 2. The signal processing block for the microphone signals can be part of the device that the microphone is built into, or it can be part of a separate device. It can also be part of cloud based processing.

[0180] Furthermore, subsystem 2 includes signal processing of the recorded tracking data. This includes frequency transformation and / or time domain based processing. Furthermore, it also includes methods to enhance the technical quality of the signal by employing noise suppression, smoothing, interpolation and extrapolation. Furthermore, it includes methods for deriving higher level information. This includes speed, acceleration, path direction, idle time, movement range and movement path. Furthermore, this includes predictions of recent movement paths and recent speeds. The signal processing block for the tracking signal can be part of the tracking device, or it can be part of a separate device. It may also be part of cloud based processing.

[0181] Subsystem 3 includes the extraction of processed microphone features.

[0182] The feature extraction block can be part of the user's wearable device or it can be part of a separate device. It can also be part of cloud-based processing.

[0183] Subsystems 2 and 3, for example, together implement detector 110, audio type classifier 130, and signal portion modifier 140 using their modules 111 and 121. For example, module 121 of subsystem 3 may output the result of audio classification to module 111 of subsystem 2 (feedback). For example, module 112 of subsystem 2 implements position determiner 120. In addition, in an embodiment, subsystems 2 and 3 may also implement signal generator 150, for example, by generating binaural room impulse responses and speaker signals through module 111 of subsystem 2.

[0184] Subsystem 4 includes methods and algorithms for estimating room acoustic parameters by using processed microphone signals, extracted features of microphone signals and processed tracking data. The output of this block is the room acoustic parameters as idle data, and control and changes to the parameters of microphone signal processing in subsystem 2. Machine learning block 131 can be part of the user device, or it can be part of a separate device. It can also be part of cloud-based processing.

[0185] In addition, subsystem 4 includes post-processing of the room acoustic idle data parameters (e.g., in block 132). This includes detecting outliers, combining individual parameters into new parameters, smoothing, extrapolation, interpolation, and plausibility verification. This block also obtains information from subsystem 2. This includes the recent position of the user in the room in order to estimate recent acoustic parameters. This block can be part of the user's device, or it can be part of a separate device. It can also be part of cloud-based processing.

[0186] Subsystem 5 includes storage and distribution of room acoustic parameters to downstream systems (e.g., in memory 141). Distribution of parameters may be accomplished immediately, and / or time responses may be stored. Storage may be performed in a device located on or near the user, or may be performed in a cloud-based system.

[0187] Use cases of embodiments of the present invention are described below.

[0188] The use case of an embodiment is home entertainment and involves a user in a home environment. For example, the user wants to focus on certain reproduction devices such as a TV, radio, PC, tablet, and wants to suppress other sources of interference (other users' devices, or children, construction noise, street noise). In this case, the user is located in the vicinity of the preferred reproduction device and selects the device or its location. Regardless of the user's location, the selected device or sound source location is acoustically emphasized until the user cancels his / her selection.

[0189] For example, the user moves near a target sound source. The user selects the target sound source via an appropriate interface, and the hearing device adjusts the audio reproduction accordingly based on the user position, the user viewing direction, and the target sound source so that the target sound source can be better heard even in the presence of interfering noise.

[0190] Alternatively, the user moves near a particularly disturbing sound source. The user selects the disturbing sound source via an appropriate interface, and the hearable device (hearing device) adjusts the audio reproduction accordingly based on the user position, the user viewing direction and the disturbing sound source so as to explicitly tune out the disturbing sound source.

[0191] Another use case for another embodiment is a cocktail party where the user is located among several speakers.

[0192] In situations where there are many speakers, for example, the user wishes to focus on one (or several) of them and wants to tune out or attenuate other interfering sound sources. In this use case, control of the hearable device should require only minimal interaction from the user. Controlling the strength of the selectivity based on bio-signals or detectable indications of difficulty in the conversation (common problems, foreign language, strong dialect) would be optional.

[0193] For example, the speakers are randomly distributed and move relative to the listener. In addition, there are periodic pauses in speech, new speakers are added or other speakers leave the scene. Possibly, interfering sounds such as music are quite loud. The selected speaker is recognized again after a pause in speech, changing his / her position or posture.

[0194] For example, the hearable device identifies speakers near the user. With appropriate control possibilities (e.g., viewing direction, attention control), the user can select a favorite speaker. The hearable device adjusts the audio reproduction based on the user's viewing direction and the selected target sound source so that the target sound source can be heard well even in the presence of interfering noise.

[0195] Alternatively, if a (previously) disliked speaker contacts the user directly, he / she must at least be audible in order to ensure a natural communication.

[0196] Another use case of another embodiment is in a motor vehicle, where the user is located in (or in) his / her motor vehicle. During driving, the user wishes to actively direct his / her auditory attention to certain reproduction devices, such as a navigation device, a radio or a conversation partner, in order to be able to hear them better next to interfering noises (wind, motor, passengers).

[0197] For example, a user and a target sound source are located at fixed positions within a motor vehicle. The user is stationary relative to the reference system, however, the vehicle itself is in motion. This requires an adapted tracking solution. The selected sound source position is acoustically emphasized until the user deselects it or until a warning signal interrupts the device's functionality.

[0198] For example, a user enters a motor vehicle and the device detects the surroundings. With appropriate control possibilities (e.g., speed recognition), the user can switch between target sound sources and the hearable device adjusts the audio reproduction according to the user's viewing direction and the selected target sound source so that the target sound source can be heard well even in the presence of interfering noise.

[0199] Alternatively, for example, a traffic-related warning signal interrupts the normal flow and cancels the user's selection. A restart of the normal flow is then performed.

[0200] Another use case of another embodiment is live music and guests involved in live music events. For example, a guest at a concert or live music performance wishes to increase attention to the performance with the aid of an audible device and wishes to tune out other guests whose behavior is disruptive. Furthermore, the audio signal itself may be optimized, for example, to balance out unfavorable listening positions or room acoustics.

[0201] For example, the user is located among many distracting sound sources; however, in most cases the performance is relatively loud. The target sound sources are located at fixed positions or at least in a defined area, however, the user can be very flexible (for example, the user may be dancing). The selected sound source positions are acoustically emphasized until the user deselects or until a warning signal interrupts the function of the device.

[0202] For example, the user selects a stage area or a musician as a target sound source. With appropriate control possibilities, the user can define the position of the stage / musician, and the hearable device adjusts the audio reproduction according to the user's viewing direction and the selected target sound source so that the target sound source can be heard well even in the presence of interfering noise.

[0203] Alternatively, for example, warning messages (e.g., evacuation, impending thunderstorm in case of outdoor activities) and warning signals can interrupt the normal flow and cancel the user's selection. Afterwards, the normal flow will be restarted.

[0204] Another use case of another embodiment is at a major event and involving guests at a major event. Thus, at a large event (e.g., a football stadium, an ice hockey arena, a large concert hall, etc.), the hearable device can be used to emphasize the voices of family and friends that would otherwise be drowned out by the noise of the crowd.

[0205] For example, a large event with many attendees at a stadium or large concert hall. A group of people (family, friends, school class) attends the event and is outside the event location or inside the event location where there is a large group of people moving around. One or several children lose eye contact with the group and call out to the group despite the loud noise. The user then turns off voice recognition and the hearable device no longer amplifies the sound.

[0206] For example, one of the group selects the sound of the missing child on the hearing device. The hearing device locates the sound. The hearing device then amplifies the sound, and the user can find the missing child (faster) based on the amplified sound.

[0207] Alternatively, for example, the missing child also wears an audible device and selects the voice of his / her parents. The audible device amplifies the voice of the parents. By amplification, the child can then find his / her parents. Thus, the child can walk back to his / her parents. Alternatively, for example, the missing child also wears an audible device and selects the voice of his / her parents. The audible device locates the voice of the parents, and the audible device notifies the distance of the voice. In this way, the child can find his / her parents more easily. Optionally, a reproduction of an artificial sound from the audible device can be provided for notification of the distance.

[0208] For example, coupling of a hearing device for selectively amplifying sound is provided, and sound profiles are stored.

[0209] Another use case of another embodiment is recreational sports and involves recreational athletes. Listening to music during sports is popular; however, it also brings dangers. Warning signals or other road users may not be heard. In addition to the reproduction of music, the audible device can also react to warning signals or shouting and temporarily interrupt the music reproduction. Another use case in this context is sports in groups. The audible devices of the sports group can be connected to ensure good communication during sports while suppressing other interfering noises.

[0210] For example, the user is mobile and possible warning signals are overlapped by many interfering sound sources. The problem is that not all warning signals may be relevant to the user (remote sirens in cities, horns on the streets). Therefore, the audible device automatically stops the music reproduction and acoustically emphasizes the communication partner's warning signal until the user cancels the selection. Subsequently, the music is reproduced normally.

[0211] For example, a user is playing sports and listening to music through an audible device. A warning signal or shouting sound related to the user is automatically detected, and the audible device interrupts the reproduction of the music. The audible device adjusts the audio reproduction to be able to hear the target sound source / acoustic environment well. The audible device then continues the reproduction of the music automatically or upon request of the user (e.g., after the warning signal ends).

[0212] Alternatively, for example, a group of athletes can connect their hearable devices. Speech intelligibility between group members is optimized, and other interfering noises are suppressed.

[0213] Another use case of another embodiment is snoring suppression and concerns all those who wish to sleep but are disturbed by snoring. People whose partners snore are disturbed during night rest and have sleeping problems. Hearables can provide relief as it can suppress the snoring sound, ensure night rest, and provide family tranquility. At the same time, the hearables let other sounds (baby crying, alarms, etc.) through, so that the user is not completely isolated acoustically from the outside world. For example, snoring detection is provided.

[0214] For example, a user has sleeping problems due to snoring noises. By using a hearable device, the user can then sleep better again, which has the effect of reducing stress.

[0215] For example, a user wears a hearable device during sleep. He / she switches the hearable device to a sleep mode that suppresses all snoring sounds. After falling asleep, he / she switches the hearable device off again.

[0216] Alternatively, other sounds such as construction noise, lawn mower noise, etc. may be suppressed during sleep.

[0217] Another use case of another embodiment is a diagnostic device for users in everyday life. The hearing device records preferences (e.g. which sound sources are selected, which attenuation / amplification is selected) and creates a profile with trends via duration of use. This data allows conclusions to be drawn about changes in hearing abilities. The aim is to detect hearing loss as early as possible.

[0218] For example, the user carries the device for several months or years in his / her daily life or in the mentioned use cases. The hearable device creates an analysis based on the selected settings and outputs warnings and suggestions to the user.

[0219] For example, a user wears a hearable device for a long period of time (months to years). The device performs analysis based on hearing preferences and outputs recommendations and warnings in case of hearing loss.

[0220] Another use case of another embodiment is a therapeutic device and involves hearing-impaired users in everyday life. As a transition device to hearing devices, helping potential patients as early as possible and thus preventive treatment of dementia. Other possibilities are use as an attention trainer (e.g. for ADHS), treatment of tinnitus and stress relief.

[0221] For example, the listener has hearing problems or attention deficits and temporarily / temporarily uses the hearable device as a hearing device. Depending on the hearing problem, it can be alleviated by the hearable device, for example by: amplification of all signals (auditory difficulties), high selectivity for preferred sound sources (attention deficit), reproduction of therapeutic sounds (tinnitus treatment).

[0222] The user, independently or with the advice of a physician, selects a treatment modality and makes preferred adjustments, and the hearable device performs the selected treatment.

[0223] Alternatively, the hearable device detects hearing problems according to the UC-PRO1, and the hearable device automatically adjusts the reproduction based on the detected problem and notifies the user.

[0224] Another use case of another embodiment is work in the public sector, involving employees in the public sector. Employees in the public sector (hospitals, pediatricians, airport counters, educators, catering, service counters, etc.) who are exposed to high intensity noise during their work wear a hearable device to emphasize the voice of one or only a few people, for example to communicate better and provide better job security by reducing stress.

[0225] For example, employees are subjected to high-intensity noise in their work environment and must talk to clients, patients, or colleagues despite background noise without being able to switch to a quieter environment. Hospital employees are subjected to high-intensity noise due to sounds and beeps from medical equipment (or any other work-related noise) and must still be able to communicate with patients or colleagues. Pediatricians and educators work amidst the noise or shouting of children and must be able to talk to parents. At an airport counter, employees have difficulty hearing airline passengers amidst the noise in the airport hall. In a busy restaurant, waiters have difficulty hearing customers' orders amidst the noise. Then, for example, the user turns off the sound selection and the hearable device no longer amplifies the sound.

[0226] For example, a person turns on an installed hearing device. The user sets the hearing device to select the sound of a nearby language, and the hearing device amplifies the nearest sound or several nearby sounds and suppresses background noise at the same time. The user then hears the relevant sound better.

[0227] Alternatively, a person sets the hearable device to continuous noise suppression. The user turns on the feature to detect available sound, which is then amplified. Thus, the user can continue to work at a lower level of noise. When the immediate vicinity of x meters is addressed, the hearable device then amplifies the sound. Thus, the user can talk to others at a low level of noise. After the conversation, the hearable device switches back to noise suppression mode, and after working, the user turns the hearable device off again.

[0228] Another use case of another embodiment is the transportation of passengers and involves users in a motor vehicle used to transport passengers. For example, users and drivers of passenger transporters want to be distracted as little as possible by passengers during driving. Although passengers are the main source of distracting sounds, it is necessary to communicate with them from time to time.

[0229] For example, the user or driver and the interference source are located at fixed positions inside a motor vehicle. The user is stationary relative to the reference system, however, the vehicle itself is moving. This requires an adapted tracking solution. Therefore, the voices and conversations of the passengers are acoustically suppressed by default unless communication is to take place.

[0230] For example, the hearable device suppresses disturbing noise for passengers by default. The user can manually cancel the suppression via an appropriate control option (voice recognition, button in the vehicle). Here, the hearable device adjusts the audio reproduction according to the selection.

[0231] Alternatively, the hearable device detects that the passenger is actively talking to the driver and temporarily deactivates noise suppression.

[0232] Another use case of another embodiment is school and education, involving a teacher and students in a classroom. In the example, the hearable device has two roles, where the functions of the devices are partially coupled. The teacher / speaker's device suppresses interfering noise and amplifies speech / questions from students. In addition, the listener's hearable device can be controlled by the teacher's device. Therefore, particularly important content can be emphasized without having to speak loudly. Students can set up their hearable devices to be able to better hear the teacher and exclude disruptive classmates.

[0233] For example, the teacher and the student are located within a defined area in a closed space (this is the rule). If all devices are coupled to each other, the relative positions are interchangeable, which in turn simplifies source separation. The selected sound source is acoustically emphasized until the user (teacher / student) deselects it, or until a warning signal interrupts the functioning of the device.

[0234] For example, a teacher or speaker presents content and the device suppresses interfering noise. The teacher wants to hear a student's question and changes the focus of the audible device to the person with the question (automatically or via appropriate control possibilities). After the exchange, all sounds are suppressed again. Furthermore, it can be provided that, for example, students who feel disturbed by classmates tune these classmates out acoustically. Furthermore, for example, students who sit far from the teacher can amplify the teacher's voice.

[0235] Alternatively, for example, the teacher and student's devices can be coupled. The selectivity of the student's device can be temporarily controlled via the teacher's device. In the case where the content is particularly important, the teacher changes the selectivity of the student's device to amplify his / her voice.

[0236] Another use case of another embodiment is the military and involves soldiers. Verbal communication between soldiers on the battlefield is carried out via radio on the one hand and shouting and direct contact on the other hand. If communication is to be carried out between different units and subgroups, radio is mostly used. Predetermined radio etiquette is usually used. Shouting and direct contact mostly occur in communication within a squad or group. During the soldier's mission, difficult acoustic conditions may occur (e.g., screaming people, noise of weapons, bad weather), which may impair both communication paths. Radio devices with headphones are usually part of the soldier's equipment. In addition to the purpose of audio reproduction, they also provide protection against greater levels of sound pressure. These devices are usually equipped with microphones to transmit ambient signals to the carrier's ears. Active noise suppression is also part of this type of system. Enhancement / extension of the range of functions enables soldiers to shout and direct contact in a noisy environment through intelligent attenuation of interfering noise and selective emphasis of speech with directional reproduction. For this purpose, the relative position of the soldier in the room / field must be known. In addition, the voice signal and the interfering noise must be separated from each other spatially and in terms of content. The system must also be able to handle high SNR levels from whispers to screams and explosions. The advantages of such a system are as follows: verbal communication between soldiers in a noisy environment, maintenance of hearing protection, the possibility of waiving radio etiquette, interception safety (since it is not a radio solution).

[0237] For example, shouting and direct contact between soldiers on a mission can be complicated by interfering noise. This problem is currently addressed by radio solutions for both near-field and greater distances. The new system enables shouting and direct contact in the near-field through intelligent spatial emphasis on individual talkers and attenuation of surrounding noise.

[0238] For example, a soldier is performing a mission. Shouts and speech are automatically detected, and the system amplifies them by simultaneously attenuating background noise. The system adjusts the spatial audio reproduction so that the target sound source can be heard well.

[0239] Alternatively, for example, the system may know the soldiers of a group and only allow audio signals from those group members to pass through.

[0240] Another use case of another embodiment relates to security personnel and guards. Thus, for example, the audible device can be used at confusing major events (celebrations, protests) to preemptively detect crimes. The selectivity of the audible device is controlled by keywords (e.g., call for help or resort to violence). This presupposes content analysis of the audio signal (e.g., speech recognition).

[0241] For example, a security guard is surrounded by many loud sound sources where the security guard and all sound sources may be moving. Under normal hearing conditions, the sound of someone calling for help cannot be heard or is heard only to a limited extent (poor SNR). Manually or automatically selected sound sources are acoustically emphasized until deselected by the user. Optionally, virtual sound objects are placed at the location / direction of the sound source of interest so that the location can be easily found (e.g., for a one-time call for help situation).

[0242] For example, the hearable device detects a sound source that is a potential source of danger. The security guard selects the sound source or event he / she wishes to follow (e.g., by selecting it on a tablet). The hearable device then adjusts the audio reproduction so that the sound source can be heard and localized well even in the presence of interfering noise.

[0243] Alternatively, for example, if the target sound source is silent, a localization signal may be placed towards / within the distance of the source.

[0244] Another use case of another embodiment is communication on stage and involving musicians. On stage, in rehearsals or concerts (e.g. band, orchestra, choir, musical), due to difficult acoustic conditions, the sounds of individual instruments (groups) may not be heard, even if they can still be heard in other environments. This impairs the interaction, because important (accompanying) sounds are no longer perceptible. The audible device can emphasize these sounds and make them audible again, and thus the interaction of the individual musicians can be improved or ensured. By using, for example, by attenuating the sound of drums, the noise exposure of the individual musicians can be reduced and hearing losses can be prevented, and the musicians can hear everything important at the same time.

[0245] For example, a musician without an audible device no longer hears at least one other sound on stage. In this case, the audible device can be used. After a rehearsal or concert, the user removes the audible device after turning it off.

[0246] In the example, the user turns on the hearing device. The user selects one or more desired instruments to be amplified. When making music together, the selected instruments are amplified and thus heard again by the hearing device. After making music, the user turns off the hearing device again.

[0247] In another example, the user turns on the listening device. The user selects a desired instrument whose volume needs to be lowered. When making music together, the listening device can lower the volume of the selected instrument so that the user can only hear it at a moderate volume.

[0248] For example, an instrument profile may be stored in an audible device.

[0249] Another use case of another embodiment is to have source separation as a software module for hearing devices in an ecosystem sense and involving manufacturers of hearing devices or users of hearing devices. Manufacturers can use source separation as an additional tool for their hearing devices and can offer it to their customers. Thus, hearing devices can also benefit from the development. Licensing models for other markets / devices (headphones, mobile phones, etc.) are also conceivable.

[0250] For example, it is difficult for a user of a hearing device to separate different sources in complex listening situations, e.g. to focus on a certain speaker. In order to be able to perform selective listening without an external additional system (e.g., signal transmission from a mobile radio via Bluetooth, selective signal transmission in a classroom via an FM device or an inductive hearing device), the user uses a hearing device with an additional function of selective hearing. Thus, even without external efforts, the user can focus on individual sources by source separation. Finally, the user turns off the additional function and continues to listen normally using the hearing device.

[0251] For example, a hearing device user acquires a new hearing device with an integrated additional feature for selective hearing. The user sets up the feature for selective hearing on the hearing device. The user then selects a profile (e.g., amplifies the loudest / closest source, voice recognition that amplifies certain sounds of one's surroundings). The hearing device amplifies the individual sources according to the set profile, while suppressing background noise when needed, and the user of the hearing device hears the individual sources from a complex auditory scene, not just the "noise" / cluttered sound sources.

[0252] Alternatively, the hearing device user acquires the additional functionality for selective hearing as software for his / her own hearing device, etc. The user installs the additional functionality for his / her hearing device. The user then sets up the functionality for selective hearing on the hearing device. The user selects a profile (amplify the loudest / closest source, voice recognition that amplifies certain sounds from one's surroundings), and the hearing device amplifies the various sources according to the set profile, as well as suppressing background noise at the same time if needed. In this case, the hearing device user hears the various sound sources from a complex auditory scene, not just the "noise" / cluttered sound sources.

[0253] For example, hearable devices may provide sound profiles that can be stored.

[0254] Another use case of another embodiment is professional sports and involves athletes in competitions. In sports such as biathlon, triathlon, cycling, marathon, etc., professional athletes rely on information from their coaches or communication with teammates. However, there are also cases where they want to protect themselves from loud sounds (shooting in biathlon, loud cheering, party horns, etc.) in order to be able to concentrate. The audible device can be adjusted for each sport / athlete to be able to select the relevant sound sources fully automatically (detection of certain sounds, volume limitation of typical interfering noises).

[0255] For example, the user can be very flexible and the type of interfering noise depends on the sport. Due to the intense physical strain, the athlete cannot or only to a limited extent control the equipment. However, in most sports there is a predetermined program (biathlon: running, shooting) and important communication partners (coach, teammates) can be defined in advance. Noise is usually suppressed or at certain stages of the activity. Communication between the athlete and teammates and coach is always emphasized.

[0256] For example, athletes use hearables that are tuned specifically for the type of sport they play. The hearables fully automatically (pre-tuned) suppress distracting noises, especially when the type of sport requires high concentration. In addition, the hearables fully automatically (pre-tuned) emphasize coaches and team members when they are within hearing range.

[0257] Another use case of another embodiment is auditory training and concerns music students, professional musicians, hobby musicians. For music rehearsals (e.g. in an orchestra, band, ensemble, music lessons), the audible device is selectively used in order to be able to track individual sounds in a filtered manner. Especially at the beginning of a rehearsal, it is helpful to listen to the final recording of the piece of music and track one's own voice. Depending on the composition, the sounds in the background cannot be heard well, as one can only hear the sounds in the foreground. With the audible device, sounds can be selectively emphasized based on instruments, etc., so that practice can be more targeted.

[0258] (Aspiring) music students can also use listenable devices to train their listening abilities in order to selectively prepare for entrance exams by gradually minimizing individual emphasis until they can eventually extract individual sounds from complex pieces without assistance.

[0259] Another possible use case is karaoke, for example, if there is no Singstar nearby, etc. The singing voice can be suppressed from a piece of music when needed so that only the instrumental version of the karaoke can be heard.

[0260] For example, a musician starts learning sounds from a musical piece. He / she listens to a recording of the piece of music through a CD player or any other reproduction medium. If the user finishes the practice, he / she closes the audible device again.

[0261] In the example, the user turns on the listening device. He / she selects the desired instrument to be amplified. While listening to the music, the listening device amplifies the sound of the instrument, lowers the volume of the remaining instruments, and the user can thus better track his / her own voice.

[0262] In another example, the user turns on the listening device. He / she selects the desired instrument to be suppressed. While listening to the music, the sound of the selected music is suppressed so that only the remaining sounds can be heard. The user can then practice the sound on his / her instrument with the other sounds without being distracted by the sounds in the recording.

[0263] In an example, the hearable device may provide a stored instrument profile.

[0264] Another use case of another embodiment is work safety and involves workers in loud environments. Workers in loud environments such as machine shops or construction sites must protect themselves from noise, but they must also be able to perceive warning signs and communicate with co-workers.

[0265] For example, the user is in a very loud environment and the target sound source (warning signal, colleague) may be significantly softer than the interfering noise. The user may be mobile; however, the interfering noise is usually stationary. As with hearing protection, the noise is permanently reduced and the hearable device automatically emphasizes the warning signal. The amplification of the speaker's sound source ensures that communication with colleagues is carried out.

[0266] For example, a user is at work and uses the audible device as hearing protection. A warning signal (e.g., a fire alarm) is acoustically emphasized and, if necessary, the user stops his / her work.

[0267] Alternatively, for example, the user is at work and uses the hearing device as hearing protection. If communication with a colleague is required, the communication partner is selected and acoustically emphasized with the aid of a suitable interface (here, for example: eye control).

[0268] Another use case of another embodiment is to use source separation as a software module for a live translator and involve a user of the live translator. The live translator translates a spoken foreign language in real time and can benefit from an upstream software module for performing source separation. Especially in the presence of multiple speakers, the software module can extract the target speaker and potentially improve the translation.

[0269] For example, the software module is part of a real-time translator (dedicated device or app on a smartphone). The user can select the target speaker, for example, via the display of the device. Advantageously, the user and the target sound source do not move or only move a little during the translation. The selected sound source position is acoustically emphasized and the translation can thus be improved.

[0270] For example, a user wishes to have a conversation in a foreign language or wishes to listen to a speaker using a foreign language. The user selects a target speaker via an appropriate interface (e.g., a GUI on a display screen), and the software module optimizes the audio recording for further use by the translator.

[0271] Another use case of another embodiment is the work safety of rescue forces, and involves firefighters, civil defense, police forces, emergency services. For rescue forces, good communication is essential for the successful execution of the mission. Despite the loud surrounding noise, it is usually impossible for rescue forces to carry out hearing protection, because this makes communication impossible. For example, firefighters must accurately convey orders and be able to hear them clearly, for example despite the loud motor sounds generated by the radio part. Therefore, the rescue forces are subject to a large noise exposure, in which hearing protection regulations cannot be followed. On the one hand, the audible device will provide hearing protection for the rescue forces, while on the other hand, communication between the rescue forces will still be enabled. In addition, with the help of the audible device, the rescue forces will not be acoustically separated from the environment when carrying helmets / protective equipment, so better support can be provided. They can communicate better and are also able to better estimate their own danger (for example, hear the type of fire that is happening).

[0272] For example, a user is subject to strong ambient noise and therefore cannot wear hearing protection, but still must be able to communicate with others. He / she uses a hearable device. After the task is completed or the danger is over, the user takes off the hearable device again.

[0273] For example, a user wears a hearable device while performing a task. He / she turns on the hearable device. The hearable device suppresses ambient noise and amplifies the speech of nearby co-workers and other speakers (e.g., fire victims).

[0274] Alternatively, the user wears the hearable device during the task. He / she turns on the hearable device, and the hearable device suppresses the surrounding noise and amplifies the colleague's speech through the radio.

[0275] Where applicable, the hearable device is specifically designed to meet structural suitability for operation in accordance with the operating specifications. Possibly, the hearable device includes an interface to a radio device.

[0276] Even if some aspects have been described in the context of an apparatus, it should be understood that the aspects also represent a description of the corresponding method, so that blocks or structural components of the apparatus are also understood as corresponding method steps or features of method steps. Similarly, aspects described or described as method steps within the context of method steps also represent a description of corresponding blocks or details or features of the corresponding apparatus. Some or all of the method steps may be performed when using a hardware device (e.g., a microprocessor, a programmable computer, or an electronic circuit). In some embodiments, some or several of the most important method steps may be performed by such a device.

[0277] Depending on the specific implementation requirements, embodiments of the present invention may be implemented in hardware or software. Various implementations may be implemented while using a digital storage medium such as a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM or flash memory, hard disk or any other magnetic or optical storage having stored thereon an electrically readable control signal that may cooperate or collaborate with a programmable computer system to cause the execution of the corresponding method. Thus, the digital storage medium may be computer readable.

[0278] Therefore, some embodiments according to the invention comprise a data carrier comprising electronically readable control signals which are capable of cooperating with a programmable computer system, such that any of the methods described herein is performed.

[0279] Generally speaking, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being for performing any method when the computer program product runs on a computer.

[0280] The program code may also be stored, for example, on a machine readable carrier.

[0281] Other embodiments include a computer program for executing any method described herein, the computer program being stored on a machine readable carrier. In other words, an embodiment of the inventive method is thus a computer program having a program code, the program code being used to execute any method described herein when the computer program is run on a computer.

[0282] Another embodiment of the inventive method is therefore a data carrier (or a digital storage medium or a computer-readable medium) on which is recorded the computer program for performing any of the methods described herein. The data carrier, the digital storage medium or the recorded medium are generally tangible and / or non-volatile.

[0283] Another embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing any of the methods described herein.The data stream or the sequence of signals may be configured to be sent, for example, via a data communication link (eg, via the Internet).

[0284] Another embodiment comprises a processing unit, for example a computer or a programmable logic device, configured to or adapted to perform any of the methods described herein.

[0285] Another embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0286] Another embodiment according to the present invention includes a device or system configured to send a computer program for performing at least one method described herein to a receiver. For example, the transmission can be electronic or optical. For example, the receiver can be a computer, a mobile device, a memory device, or the like. For example, the device or system can include a file server for sending the computer program to the receiver.

[0287] In some embodiments, a programmable logic device (e.g., a field programmable gate array FPGA) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor to perform any method described herein. Generally speaking, in some embodiments, the method is performed by any hardware device. The hardware device can be any general-purpose hardware, such as a computer processor (CPU), or can be method-specific hardware, such as an ASIC.

[0288] The above embodiments are merely illustrative of the principles of the present invention. It should be understood that other persons skilled in the art will appreciate modifications and variations to the arrangements and details described herein. Therefore, the present invention is intended to be limited only by the scope of the appended claims, and not by the specific details presented herein through the description and discussion of the embodiments.

[0289] References

[0290] [1]V. Valimaki, A. Franck, J. Ramo, H. Gamper, and L. Savioja, "Assisted listening using a headset: Enhancing audio perception in real, augmented, and virtual environments," IEEE Signal Processing Magazine, volume32, no. 2, pp. 92–99, March 2015.

[0291] [2]K. Brandenburg, E. Cano, F. Klein, T. Köllmer, H. Lukashevich, A.Neidhardt, U. Sloma, and S. Werner, “Plausible augmentation of auditoryscenes using dynamic binaural synthesis for personalized auditory realities,”in Proc. of AES International Conference on Audio for Virtual and AugmentedReality, August 2018.

[0292] [3]S. Argentieri, P. Dans, and P. Soures, “A survey on sound sourcelocalization in robotics: From binaural to array processing methods,”Computer Speech Language, volume 34, no. 1, pp. 87–112, 2015.

[0293] [4]D. FitzGerald, A. Liutkus, and R. Badeau, “Projection-baseddemixing of spatial audio,” IEEE / ACM Trans. on Audio, Speech, and LanguageProcessing, volume 24, no. 9, pp. 1560–1572, 2016.

[0294] [5]E. Cano, D. FitzGerald, A. Liutkus, M. D. Plumbley, and F. Stöter,“Musical source separation: An introduction,” IEEE Signal ProcessingMagazine, volume 36, no. 1, pp. 31–40, January 2019.

[0295] [6]S. Gannot, E. Vincent, S. Markovich-Golan, and A. Ozerov, “Aconsolidated perspective on multimicrophone speech enhancement and sourceseparation,” IEEE / ACM Transactions on Audio, Speech, and Language Processing,volume 25, no. 4, pp. 692–730, April 2017.

[0296] [7]E. Cano, J. Nowak, and S. Grollmisch, “Exploring sound sourceseparation for acoustic condition monitoring in industrial scenarios,” inProc. of 25th European Signal Processing Conference (EUSIPCO), August 2017,pp. 2264–2268.

[0297] [8]T. Gerkmann, M. Krawczyk-Becker, and J. Le Roux, “Phase processingfor single-channel speech enhancement: History and recent advances,” IEEESignal Processing Magazine, volume 32, no. 2, pp. 55–66, March 2015.

[0298] [9]E. Vincent, T. Virtanen, and S. Gannot, Audio Source Separationand Speech Enhancement. Wiley, 2018.

[0299]

[10] D. Matz, E. Cano, and J. Abeßer, “New sonorities for early jazzrecordings using sound source separation and automatic mixing tools,” inProc. of the 16th International Society for Music Information RetrievalConference. Malaga, Spain: ISMIR, October 2015, pp. 749–755.

[0300]

[11] S. M. Kuo and D. R. Morgan, “Active noise control: a tutorialreview,” Proceedings of the IEEE, volume 87, no. 6, pp. 943–973, June 1999.

[0301]

[12] A. McPherson, R. Jack, and G. Moro, “Action-sound latency: Areour tools fast enough?” in Proceedings of the International Conference on NewInterfaces for Musical Expression, July 2016.

[0302]

[13] C. Rottondi, C. Chafe, C. Allocchio, and A. Sarti, “An overviewon networked music performance technologies,” IEEE Access, volume 4, pp.8823–8843, 2016.

[0303]

[14] S. Liebich, J. Fabry, P. Jax, and P. Vary, “Signal processingchallenges for active noise cancellation headphones,” in SpeechCommunication; 13th ITG-Symposium, October 2018, pp. 1–5.

[0304]

[15] E. Cano, J. Liebetrau, D. Fitzgerald, and K. Brandenburg, “Thedimensions of perceptual quality of sound source separation,” in Proc. ofIEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), April 2018, pp. 601–605.

[0305]

[16] P. M. Delgado and J. Herre, “Objective assessment of spatialaudio quality using directional loudness maps,” in Proc. of IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP),May 2019, pp. 621–625.

[0306]

[17] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “Analgorithm for intelligibility prediction of time-frequency weighted noisyspeech,” IEEE Transactions on Audio, Speech, and Language Processing, volume19, no. 7, pp. 2125–2136, September 2011.

[0307]

[18] M. D. Plumbley, C. Kroos, J. P. Bello, G. Richard, D. P. Ellis,and A. Mesaros, Proceedings of the Detection and Classification of AcousticScenes and Events 2018 Workshop (DCASE2018). Tampere University ofTechnology. Laboratory of Signal Processing, 2018.

[0308]

[19] R. Serizel, N. Turpault, H. Eghbal-Zadeh, and A. Parag Shah,“Large- Scale Weakly Labeled Semi-Supervised Sound Event Detection inDomestic Environments,” July 2018, submitted to DCASE2018 Workshop.

[0309]

[20] L. JiaKai, “Mean teacher convolution system for dcase 2018 task4,” DCASE2018 Challenge, Tech. Rep., September 2018.

[0310]

[21] G. Parascandolo, H. Huttunen, and T. Virtanen, “Recurrent neuralnetworks for polyphonic sound event detection in real life recordings,” inProc. of IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP), March 2016, pp. 6440–6444.

[0311]

[22] E. C¸ Çakir and T. Virtanen, “End-to-end polyphonic sound eventdetection using convolutional recurrent neural networks with learned time-frequency representation input,” in Proc. of International Joint Conferenceon Neural Networks (IJCNN), July 2018, pp. 1–7.

[0312]

[23] Y. Xu, Q. Kong, W. Wang, and M. D. Plumbley, “Large-Scale WeaklySupervised Audio Classification Using Gated Convolutional Neural Network,” inProceedings of the IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP), Calgary, AB, Canada, 2018, pp. 121–125.

[0313]

[24] B. Frenay and M. Verleysen, “Classification in the presence oflabel noise: A survey,” IEEE Transactions on Neural Networks and LearningSystems, volume 25, no. 5, pp. 845–869, May 2014.

[0314]

[25] E. Fonseca, M. Plakal, D. P. W. Ellis, F. Font, X. Favory, and X.Serra, “Learning sound event classifiers from web audio with noisy labels,”in Proceedings of IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP), Brighton, UK, 2019.

[0315]

[26] M. Dorfer and G. Widmer, “Training general-purpose audio taggingnetworks with noisy labels and iterative self-verification,” in Proceedingsof the Detection and Classification of Acoustic Scenes and Events 2018Workshop (DCASE2018), Surrey, UK, 2018.

[0316]

[27] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, “Soundevent localization and detection of overlapping sources using convolutionalrecurrent neural networks,” IEEE Journal of Selected Topics in SignalProcessing, pp. 1–1, 2018.

[0317]

[28] Y. Jung, Y. Kim, Y. Choi, and H. Kim, “Joint learning usingdenoising variational autoencoders for voice activity detection,” in Proc. ofInterspeech, September 2018, pp. 1210–1214.

[0318]

[29] F. Eyben, F. Weninger, S. Squartini, and B. Schuller, “Real-lifevoice activity detection with LSTM recurrent neural networks and anapplication to hollywood movies,” in Proc. of IEEE International Conferenceon Acoustics, Speech and Signal Processing, May 2013, pp. 483–487.

[0319]

[30] R. Zazo-Candil, T. N. Sainath, G. Simko, and C. Parada, “Featurelearning with raw-waveform CLDNNs for voice activity detection,” in Proc. ofINTERSPEECH, 2016.

[0320]

[31] M. McLaren, Y. Lei, and L. Ferrer, “Advances in deep neuralnetwork approaches to speaker recognition,” in Proc. of IEEE InternationalConference on Acoustics, Speech and Signal Processing (ICASSP), April 2015,pp. 4814–4818.

[0321]

[32] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur,“X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. of IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP),April 2018, pp. 5329–5333.

[0322]

[33] M. McLaren, D. Castán, M. K. Nandwana, L. Ferrer, and E. Yilmaz,“How to train your speaker embeddings extractor,” in Odyssey, 2018.

[0323]

[34] S. O. Sadjadi, J. W. Pelecanos, and S. Ganapathy, “The IBMspeaker recognition system: Recent advances and error analysis,” in Proc. ofInterspeech, 2016, pp. 3633–3637.

[0324]

[35] Y. Han, J. Kim, and K. Lee, “Deep convolutional neural networksfor predominant instrument recognition in polyphonic music,” IEEE / ACMTransactions on Audio, Speech, and Language Processing, volume 25, no. 1, pp.208–221, January 2017.

[0325]

[36] V. Lonstanlen and C.-E. Cella, “Deep convolutional networks onthe pitch spiral for musical instrument recognition,” in Proceedings of the17th International Society for Music Information Retrieval Conference. NewYork, USA: ISMIR, 2016, pp. 612–618.

[0326]

[37] S. Gururani, C. Summers, and A. Lerch, “Instrument activitydetection in polyphonic music using deep neural networks,” in Proceedings ofthe 19th International Society for Music Information Retrieval Conference.Paris, France: ISMIR, September 2018, pp. 569–576.

[0327]

[38] J. Schlütter and B. Lehner, “Zero mean convolutions for level-invariant singing voice detection,” in Proceedings of the 19th InternationalSociety for Music Information Retrieval Conference. Paris, France: ISMIR,September 2018, pp. 321–326.

[0328]

[39] S. Delikaris-Manias, D. Pavlidi, A. Mouchtaris, and V. Pulkki,“DOA estimation with histogram analysis of spatially constrained activeintensity vectors,” in Proc. of IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), March 2017, pp. 526–530.

[0329]

[40] S. Chakrabarty and E. A. P. Habets, “Multi-speaker DOA estimationusing deep convolutional networks trained with noise signals,” IEEE Journalof Selected Topics in Signal Processing, volume 13, no. 1, pp. 8– 21, March2019.

[0330]

[41] X. Li, L. Girin, R. Horaud, and S. Gannot, “Multiple-speakerlocalization based on direct-path features and likelihood maximization withspatial sparsity regularization,” IEEE / ACM Transactions on Audio, Speech, andLanguage Processing, volume 25, no. 10, pp. 1997–2012, October 2017.

[0331]

[42] F. Grondin and F. Michaud, “Lightweight and optimized soundsource localization and tracking methods for open and closed microphone arrayconfigurations,” Robotics and Autonomous Systems, volume 113, pp. 63 – 80,2019.

[0332]

[43] D. Yook, T. Lee, and Y. Cho, “Fast sound source localizationusing two-level search space clustering,” IEEE Transactions on Cybernetics,volume 46, no. 1, pp. 20–26, January 2016.

[0333]

[44] D. Pavlidi, A. Griffin, M. Puigt, and A. Mouchtaris, “Real-timemultiple sound source localization and counting using a circular microphonearray,” IEEE Transactions on Audio, Speech, and Language Processing, volume21, no. 10, pp. 2193–2206, October 2013.

[0334]

[45] P. Vecchiotti, N. Ma, S. Squartini, and G. J. Brown, “End-to-endbinaural sound localisation from the raw waveform,” in Proc. of IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP),May 2019, pp. 451–455.

[0335]

[46] Y. Luo, Z. Chen, and N. Mesgarani, “Speaker-independent speechseparation with deep attractor network,” IEEE / ACM Transactions on Audio,Speech, and Language Processing, volume 26, no. 4, pp. 787–796, April 2018.

[0336]

[47] Z. Wang, J. Le Roux, and J. R. Hershey, “Multi-channel deepclustering: Discriminative spectral and spatial embeddings for speaker-independent speech separation,” in Proc. of IEEE International Conference onAcoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 1–5.

[0337]

[48] G. Naithani, T. Barker, G. Parascandolo, L. BramslŁw, N. H.Pontoppidan, and T. Virtanen, “Low latency sound source separation usingconvolutional recurrent neural networks,” in Proc. of IEEE Workshop onApplications of Signal Processing to Audio and Acoustics (WASPAA), October2017, pp. 71–75.

[0338]

[49] M. Sunohara, C. Haruta, and N. Ono, “Low-latency real-time blindsource separation for hearing aids based on time-domain implementation ofonline independent vector analysis with truncation of non-causal components,”in Proc. of IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP), March 2017, pp. 216–220.

[0339]

[50] Y. Luo and N. Mesgarani, “TaSNet: Time-domain audio separationnetwork for real-time, single-channel speech separation,” in Proc. of IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP),April 2018, pp. 696–700.

[0340]

[51] J. Chua, G. Wang, and W. B. Kleijn, “Convolutive blind sourceseparation with low latency,” in Proc. of IEEE International Workshop onAcoustic Signal Enhancement (IWAENC), September 2016, pp. 1–5.

[0341]

[52] Z. Rafii, A. Liutkus, F. Stöter, S. I. Mimilakis, D. FitzGerald,and B. Pardo, “An overview of lead and accompaniment separation in music,”IEEE / ACM Transactions on Audio, Speech, and Language Processing, volume 26,no. 8, pp. 1307–1335, August 2018.

[0342]

[53] F.-R. Stöter, A. Liutkus, and N. Ito, “The 2018 signal separationevaluation campaign,” in Latent Variable Analysis and Signal Separation, Y.Deville, S. Gannot, R. Mason, M. D. Plumbley, and D. Ward, Eds. Cham:Springer International Publishing, 2018, pp. 293–305.

[0343]

[54] J.-L. Durrieu, B. David, and G. Richard, “A musically motivatedmidlevel representation for pitch estimation and musical audio sourceseparation,” Selected Topics in Signal Processing, IEEE Journal of, volume 5,no. 6, pp. 1180 –1191, October 2011.

[0344]

[55] S. Uhlich, M. Porcu, F. Giron, M. Enenkl, T. Kemp, N. Takahashi,and Y. Mitsufuji, “Improving music source separation based on deep neuralnetworks through data augmentation and network blending,” in Proc. of IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP),2017.

[0345]

[56] P. N. Samarasinghe, W. Zhang, and T. D. Abhayapala, “Recentadvances in active noise control inside automobile cabins: Toward quietercars,” IEEE Signal Processing Magazine, volume 33, no. 6, pp. 61–73, November2016.

[0346]

[57] S. Papini, R. L. Pinto, E. B. Medeiros, and F. B. Coelho, “Hybridapproach to noise control of industrial exhaust systems,” Applied Acoustics,volume 125, pp. 102 – 112, 2017.

[0347]

[58] J. Zhang, T. D. Abhayapala, W. Zhang, P. N. Samarasinghe, and S.Jiang, “Active noise control over space: A wave domain approach,” IEEE / ACMTransactions on Audio, Speech, and Language Processing, volume 26, no. 4, pp.774–786, April 2018.

[0348]

[59] X. Lu, Y. Tsao, S. Matsuda, and C. Hori, “Speech enhancementbased on deep denoising autoencoder,” in Proc. of Interspeech, 2013.

[0349]

[60] Y. Xu, J. Du, L. Dai, and C. Lee, “A regression approach tospeech enhancement based on deep neural networks,” IEEE / ACM Transactions onAudio, Speech, and Language Processing, volume 23, no. 1, pp. 7–19, January2015.

[0350]

[61] S. Pascual, A. Bonafonte, and J. Serrà, “SEGAN: speechenhancement generative adversarial network,” in Proc. of Interspeech, August2017, pp. 3642–3646.

[0351]

[62] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J.R. Hershey, and B. Schuller, “Speech enhancement with LSTM recurrent neuralnetworks and its application to noise-robust ASR,” in Latent VariableAnalysis and Signal Separation, E. Vincent, A. Yeredor, Z. Koldovský, and P.Tichavský, Eds. Cham: Springer International Publishing, 2015, pp. 91–99.

[0352]

[63] H. Wierstorf, D. Ward, R. Mason, E. M. Grais, C. Hummersone, andM. D. Plumbley, “Perceptual evaluation of source separation for remixingmusic,” in Proc. of Audio Engineering Society Convention 143, October 2017.

[0353]

[64] J. Pons, J. Janer, T. Rode, and W. Nogueira, “Remixing musicusing source separation algorithms to improve the musical experience ofcochlear implant users,” The Journal of the Acoustical Society of America,volume 140, no. 6, pp. 4338–4349, 2016.

[0354]

[65] Q. Kong, Y. Xu, W. Wang, and M. D. Plumbley, “A joint separation-classification model for sound event detection of weakly labelled data,” inProceedings of IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP), March 2018.

[0355]

[66] T. v. Neumann, K. Kinoshita, M. Delcroix, S. Araki, T. Nakatani,and R. Haeb-Umbach, “All-neural online source separation, counting, anddiarization for meeting analysis,” in Proc. of IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 91–95.

[0356]

[67] S. Gharib, K. Drossos, E. Cakir, D. Serdyuk, and T. Virtanen,“Unsupervised adversarial domain adaptation for acoustic sceneclassification,” in Proceedings of the Detection and Classification ofAcoustic Scenes and Events Workshop (DCASE), November 2018, pp. 138–142.

[0357]

[68] A. Mesaros, T. Heittola, and T. Virtanen, “A multi-device datasetfor urban acoustic scene classification,” in Proceedings of the Detection andClassification of Acoustic Scenes and Events Workshop, Surrey, UK, 2018.

[0358]

[69] J. Abeßer, M. Götze, S. Kühnlenz, R. Gräfe, C. Kühn, T. Clauß, H.Lukashevich, "A Distributed Sensor Network for Monitoring Noise Level andNoise Sources in Urban Environments," in Proceedings of the 6th IEEEInternational Conference on Future Internet of Things and Cloud (FiCloud),Barcelona, Spain, pp. 318-324., 2018.

[0359]

[70] T. Virtanen, M. D. Plumbley, D. Ellis (Eds.), "ComputationalAnalysis of Sound Scenes and Events," Springer, 2018.

[0360]

[71] J. Abeßer, S. Ioannis Mimilakis, R. Gräfe, H. Lukashevich, "Acoustic scene classification by combining autoencoder-based dimensionalityreduction and convolutional neural net-works," in Proceedings of the 2ndDCASE Workshop on Detection and Classification of Acoustic Scenes and Events,Munich, Germany, 2017.

[0361]

[72] A. Avni, J. Ahrens, M. Geierc, S. Spors, H. Wierstorf, B.Rafaely, “Spatial perception of sound fields recorded by spherical microphonearrays with varying spatial resolution,” Journal of the Acoustic Society ofAmerica, 133(5), pp. 2711–2721, 2013.

[0362]

[73] E. Cano, D. FitzGerald, K. Brandenburg, “Evaluation of quality ofsound source separation algorithms: Human perception vs quantitativemetrics,” in Proceedings of the 24th European Signal Processing Conference(EUSIPCO), pp. 1758–1762, 2016.

[0363]

[74] S. Marchand, “Audio scene transformation using informed sourceseparation,” The Journal of the Acoustical Society of America, 140(4), p.3091, 2016.

[0364]

[75] S. Grollmisch, J. Abeßer, J. Liebetrau, H. Lukashevich, "Soundingindustry: Challenges and datasets for industrial sound analysis (ISA)," inProceedings of the 27th European Signal Processing Conference (EUSIPCO)(submitted), A Coruna, Spain, 2019.

[0365]

[76] J. Abeßer, M. Müller, "Fundamental frequency contourclassification: A comparison between hand-crafted and CNN-based features, "in Proceedings of the 44th IEEE International Conference on Acoustics,Speech, and Signal Processing (ICASSP), 2019.

[0366]

[77] J. Abeßer, S. Balke, M. Müller, "Improving bass saliencyestimation using label propagation and transfer learning," in Proceedings ofthe 19th International Society for Music Information Retrieval Conference(ISMIR), Paris, France, pp. 306–312, 2018.

[0367]

[78] C.-R. Nagar, J. Abeßer, S. Grollmisch, "Towards CNN-basedacoustic modeling of seventh chords for recognition chord recognition," inProceedings of the 16th Sound & Music Computing Conference (SMC) (submitted),Malaga, Spain, 2019.

[0368]

[79] J. S. Gómez, J. Abeßer, E. Cano, "Jazz solo instrumentclassification with convolutional neural networks, source separation, andtransfer learning", in Proceedings of the 19th International Society forMusic Information Retrieval Conference (ISMIR), Paris, France, pp. 577– 584,2018.

[0369]

[80] J. R. Hershey, Z. Chen, J. Le Roux, S. Watanabe, "Deepclustering: Discriminative embeddings for segmentation and separation," inProceedings of the IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP), pp. 31-35, 2016.

[0370]

[81] E. Cano, G. Schuller, C. Dittmar, "Pitch-informed solo andaccompaniment separation towards its use in music education applications",EURASIP Journal on Advances in Signal Processing, 2014:23, pp. 1-19.

[0371]

[82] S. I. Mimilakis, K. Drossos, J. F. Santos, G. Schuller, T.Virtanen, Y. Bengio, "Monaural Singing Voice Separation with Skip-FilteringConnections and Recurrent Inference of Time-Frequency Mask," in Proceedingsof the IEEE International Conference on Acoustics, Speech, and SignalProcessing (ICASSP), Calgary, Canada, S.721-725, 2018.

[0372]

[83] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W.Lawrence, R. C. Moore, M. Plakal, M. Ritter, "Audio Set: An ontology andhuman-labeled dataset for audio events," in Proceedings of the IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP),New Orleans, USA, 2017.

[0373]

[84] Kleiner, M. “Acoustics and Audio Technology,”. 3rd ed. USA: J.Ross Publishing, 2012.

[0374]

[85] M. Dickreiter, V. Dittel, W. Hoeg, M. Wöhr, M. „Handbuch derTonstudiotechnik,“ A. medienakademie (Eds). 7th edition, Vol. 1., Munich:K.G. Saur Verlag, 2008.

[0375]

[86] F. Müller, M. Karau. „Transparant hearing,“ in: CHI ‚02 ExtendedAbstracts on Human Factors in Computing Systems (CHI EA ’02), Minneapolis,USA, pp. 730- 731, April 2002.

[0376]

[87] L. Vieira. "Super hearing: a study on virtual prototyping forhearables and hearing aids," Master Thesis, Aalborg University, 2018.Available: https: / / projekter.aau.dk / projekter / files / 287515943 / MasterThesis_ Luis.pdf .

[0377]

[88] Sennheiser, "AMBEO Smart Headset," [Online]. Available:

[0378] https: / / de-de.sennheiser.com / finalstop [Accessed: March 1, 2019].

[0379]

[89] Orosound "Tilde Earphones" [Online]. Available:

[0380] https: / / www.orosound.com / tilde-earphones / [Accessed; March 1, 2019].

[0381]

[90] Brandenburg, K., Cano Ceron, E., Klein, F., Köllmer, T., Lukashevich, H., Neidhardt, A., Nowak, J., Sloma, U., und Werner, S., „Personalized auditory reality,” in 44. Jahrestagung für Akustik (DAGA), Garching bei München, Deutsche Gesellschaft für Akustik (DEGA), 2018.

[0382]

[91] US 2015 195641 A1, Application date: January 6, 2014; Published on July 9, 2015.

Claims

1. A system for assisting selective hearing, the system comprising: A detector (110) for detecting audio source signal portions of one or more audio sources by using at least two received microphone signals of an auditory environment, a position determiner (120) for assigning position information to each of the one or more audio sources, an audio type classifier (130) for assigning an audio signal type to an audio source signal portion of each audio source of the one or more audio sources, a signal portion modifier (140) for modifying an audio source signal portion of at least one audio source of the one or more audio sources in dependence on an audio signal type of the audio source signal portion of the at least one audio source so as to obtain a modified audio signal portion of the at least one audio source, and A signal generator (150) is used to generate a plurality of binaural room impulse responses for each audio source according to the position information of each audio source in the one or more audio sources and the orientation of the user's head, and to generate at least two speaker signals according to the plurality of binaural room impulse responses and according to the modified audio signal portion of the at least one audio source.

2. The system according to claim 1, in, The detector (110) is configured to detect audio source signal portions of the one or more audio sources by using a deep learning model.

3. The system according to claim 1, in, The position determiner (120) is configured to determine the position information for each of the one or more audio sources based on the captured image or recorded video.

4. The system according to claim 3, in, The position determiner (120) is configured to: for each of the one or more audio sources, determine the position information based on the video by detecting lip movements of a person in the video and assigning the position information to an audio source signal portion of one of the one or more audio sources based on the lip movements.

5. The system according to claim 1, in, The detector (110) is configured to determine one or more acoustic properties of the auditory environment based on the at least two received microphone signals.

6. The system according to claim 5, in, The signal generator (150) is configured to determine the plurality of binaural room impulse responses based on one or more acoustic properties of the auditory environment.

7. The system according to claim 1, in, The signal portion modifier (140) is configured to select the at least one audio source whose audio source signal portion is modified according to a previously learned user scenario, and to modify the audio source signal portion of the at least one audio source according to the previously learned user scenario.

8. The system according to claim 7, wherein: The system includes a user interface (160) for selecting the previously learned user scenario from a group of two or more previously learned user scenarios.

9. The system according to claim 1, in, The detector (110) and / or the position determiner (120) and / or the audio type classifier (130) and / or the signal portion modifier (140) and / or the signal generator (150) are configured to perform parallel signal processing using Hough transform or employing multiple VLSI chips or by employing multiple memristors.

10. The system according to claim 1, in, The system comprises a hearing device (170) used as a hearing aid for a user with limited hearing ability and / or hearing impairment, wherein the hearing device (170) comprises at least two loudspeakers (171, 172) for outputting the at least two loudspeaker signals.

11. The system according to claim 1, in, The system comprises at least two speakers (181, 182) for outputting the at least two speaker signals, and a shell structure (183) accommodating the at least two speakers (181, 182), wherein at least one of the shell structures (183) is suitable for being fixed to a user's head or any other body part of the user.

12. The system according to claim 1, in, The system comprises a headset (180), the headset (180) comprising at least two speakers (181, 182) for outputting the at least two speaker signals.

13. The system according to claim 12, wherein: The detector (110) and the position determiner (120) and the audio type classifier (130) and the signal portion modifier (140) and the signal generator (150) are integrated into the earphone (180).

14. The system according to claim 12, in, The system comprises a remote device (190), the remote device (190) comprising the detector (110) and the position determiner (120) and the audio type classifier (130) and the signal portion modifier (140) and the signal generator (150), Wherein, the remote device is spatially separated from the headset.

15. The system of claim 14, wherein: The remote device (190) is a smartphone.

16. The system of claim 1, wherein: The system further comprises: means for determining one or more room acoustic parameters, wherein the apparatus is configured to obtain microphone data comprising one or more microphone signals, wherein the apparatus is configured to obtain tracking data regarding the position and / or orientation of a user, Therein, the apparatus is configured to determine the one or more room acoustic parameters based on the microphone data and based on the tracking data.

17. The system according to claim 16, in, The apparatus is configured to employ machine learning to determine the one or more room acoustic parameters from the microphone data and from the tracking data.

18. The system of claim 17, wherein: The apparatus is configured to employ machine learning, wherein the apparatus is configured to employ a neural network.

19. The system of claim 17, wherein: The device is configured to employ cloud-based processing for machine learning.

20. The system according to claim 16, in, The one or more room acoustic parameters include reverberation time.

21. The system of claim 16, wherein: The one or more room acoustic parameters include a direct-to-reverberation ratio.

22. The system according to claim 16, in, The tracking data includes x-coordinates, y-coordinates, and z-coordinates marking the location of the user.

23. The system according to claim 16, in, The tracking data includes pitch, yaw, and roll coordinates for marking the orientation of the user.

24. The system according to claim 16, in, The apparatus is configured to transform the one or more microphone signals from the time domain to the frequency domain, wherein the apparatus is configured to extract one or more features of the one or more microphone signals in the frequency domain, And wherein the apparatus is configured to determine the one or more room acoustic parameters based on the one or more characteristics.

25. The system according to claim 24, in, The apparatus is configured to employ cloud-based processing to extract the one or more features.

26. The system of claim 16, wherein: The device comprises a microphone arrangement with a number of microphones for recording a number of microphone signals.

27. The system according to claim 26, in, The microphone arrangement is configured to be worn on a user's body.

28. The system according to claim 16, in, The signal portion modifier (140) is configured to perform a change of an audio source signal portion of at least one of the one or more audio sources in dependence on at least one of the one or more room acoustic parameters; and / or The signal generator (150) is configured to generate at least one of the plurality of binaural room impulse responses for each of the one or more audio sources according to at least one of the one or more room acoustic parameters.

29. A method for assisting selective hearing, the method comprising: detecting audio source signal portions of one or more audio sources by using at least two received microphone signals of an auditory environment, assigning position information to each of the one or more audio sources, assigning an audio signal type to an audio source signal portion of each of the one or more audio sources, modifying the audio source signal portion of at least one audio source of the one or more audio sources in dependence on the audio signal type of the audio source signal portion of the at least one audio source so as to obtain a modified audio signal portion of the at least one audio source, and A plurality of binaural room impulse responses are generated for each audio source according to the position information of each audio source of the one or more audio sources and the orientation of the user's head, and at least two speaker signals are generated according to the plurality of binaural room impulse responses and according to the modified audio signal portion of the at least one audio source.

30. A non-transitory computer readable medium comprising computer readable instructions which, when executed by a computer or a signal processor, cause the computer or signal processor to perform the method of claim 29.

Images