Systems and methods for headphone equalization and room adaptation for binaural reproduction in augmented reality

Abstract

A system is provided that includes an analyzer (152) for determining a plurality of binaural room impulse responses and a loudspeaker signal generator (154) for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source, wherein the analyzer (152) is configured to determine the plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account effects resulting from a user wearing headphones.

Description

[Technical Field]

[0001] The present invention relates to headphone equalization and room adaptation for binaural reproduction in augmented reality (AR). [Background technology]

[0002] Selective hearing (SH) refers to the ability of a listener to direct their attention to a particular sound source or sources in an auditory scene, which in turn means that the listener's focus on sources of no interest is reduced.

[0003] Thus, human listeners can communicate in large environments, typically by exploiting different aspects: when listening with two ears, there are direction-dependent time and level differences, as well as different direction-dependent spectral coloration of sound. The latter allows auditory perception to determine the direction of sound sources and separate different sound sources, even when listening with one ear.

[0004] The exact location of the sound source cannot be determined by time and level differences alone. decision It is not enough to calculate the time difference and the level difference, and the resulting position will lie on the hyperbolic surface. decision The ambiguity is called the cone of confusion. In a room, each sound source is reflected by boundary surfaces. Each of these so-called mirror sources is located on a further hyperbolic surface. Human hearing combines information about the direct sound with the associated reflections into an auditory event, thereby resolving the confusion ambiguity. At the same time, the reflections belonging to a sound source increase the perceived loudness of the sound source.

[0005] Furthermore, in the case of natural sound sources, especially speech, signal parts of different frequencies are temporally combined. In binaural hearing, all of these aspects are used together. Furthermore, large disturbance sources that can be well localized can be, so to speak, actively ignored.

[0006] In the literature, the concept of selective hearing is related to other terms such as assisted hearing [1], virtual hearing environments and amplified hearing environments [2]. Assisted hearing is a broader term that includes virtual, amplified and SH applications.

[0007] According to the prior art, classical hearing devices operate primarily in a monophonic manner, i.e., the signal processing for the left and right ears is completely independent in terms of frequency response and dynamic compression, resulting in the loss of time, level, and frequency differences between the ear signals.

[0008] Modern so-called binaural hearing devices combine the correction coefficients of two hearing devices. They often have several microphones, but usually only the microphone with the "most speech-like" signal is selected, and no explicit beamforming is calculated. In complex hearing situations, desired and undesired sound signals are amplified equally, and therefore focusing on the desired sound component is not supported.

[0009] For example, in the field of hands-free devices for telephones, several microphones are already used today. A so-called beam is calculated from the individual microphone signals, amplifying sounds coming from the beam's direction and reducing sounds coming from other directions. Current methods train on constant background sounds (e.g., engine and wind noise in a car) and then, via additional beams, train on large, well-localized disturbances and subtract them from the active signal (e.g., generalized sidelobe cancellers). Sometimes, telephone systems use detectors that detect static characteristics of speech and suppress everything unstructured like speech. In hands-free devices, only a mono signal is ultimately transmitted, losing valuable spatial information along the transmission path to capture the situation and provide the illusion of "one was there," especially when several speakers are on a reciprocal call. Suppressing non-speech signals can result in the loss of important information about the other party's acoustic environment, potentially disrupting communication.

[0010] Humans are naturally capable of consciously focusing on "selective listening" to individual sound sources in their environment. An automated system for selective listening using artificial intelligence (AI) must first learn the underlying concepts. Automated decomposition of an acoustic scene (scene decomposition) first requires the detection and classification of all active sound sources, and then separating them so that they can be further processed, amplified, or attenuated as separate audio objects.

[0011] The research field of auditory scene analysis attempts to detect and classify temporally located sound events, such as steps, clapping, or voices, as well as more global acoustic scenes, such as concerts, restaurants, or supermarkets, based on recorded audio signals. In this case, current methods exclusively use methods from the fields of artificial intelligence (AI) and deep learning. This involves data-driven training of deep neural networks that learn based on large training sets to detect characteristic patterns within audio signals

[70] . Inspired by advances in image processing (computer vision) and speech processing (natural language processing), among others, the principle is to mix convolutional neural networks for two-dimensional pattern detection in spectrogram representations with recurrent layers (recurrent neural networks) for temporal sound modeling.

[0012] Speech analysis presents a particular set of challenges that must be addressed. Due to their complexity, deep learning models are very data-intensive. In contrast to the research fields of image processing and speech processing, the datasets available for speech processing are relatively small. The largest dataset is the AudioSet dataset from Google

[83] , which has approximately 2 million sound examples and 632 different sound event classes, and most datasets used in research are significantly smaller. This small amount of training data can be addressed, for example, using transfer learning, where models pre-trained on large datasets can then be adapted to fit the use case. decisionThe resulting new classes are then fine-tuned on a smaller dataset (fine-tuning)

[77] . Furthermore, methods from semi-supervised learning are often used to include unannotated audio data, which is commonly available in large quantities, in training.

[0013] A further important difference compared to image processing is that in the case of simultaneously heard sound events, there is no masking of sound objects (as in the case of images) but rather a complex phase-dependent overlap. Current algorithms in deep learning use so-called "attention" mechanisms, which allow, for example, the model to focus classification on specific time segments or frequency ranges

[23] . The detection of sound events is further complicated by the high variance regarding their duration. Algorithms should be able to robustly detect very short events, such as gunshots, as well as long events, such as passing trains.

[0014] Due to the strong dependence of models on the acoustic conditions in the recording of the training data, models often exhibit unexpected behavior in new acoustic environments, e.g., those that differ in terms of spatial reverberation or microphone positioning. Various solutions have been developed to alleviate this problem. For example, data augmentation methods attempt to achieve greater robustness and invariance of models through the simulation of different acoustic conditions

[68] and the artificial overlap of different sound sources. Furthermore, the parameters of complex neural networks can be adjusted in different ways to avoid overtraining and specialization on the training data while simultaneously achieving better generalization to unseen data. In recent years, different algorithms have been proposed for "domain adaptation"

[67] to adapt previously trained models to new application conditions. For the in-headphone usage scenario planned in this project, the real-time capability of sound source detection algorithms is fundamentally important. Here, a trade-off must be made between the complexity of the neural network and the maximum possible number of computational operations on the underlying computing platform. Even if sound events have a long duration, they must be detected as quickly as possible to initiate corresponding sound source separation.

[0015] At the Fraunhofer IDMT, much research has been conducted in recent years in the field of automatic sound source detection. In the research project "Stadtram," a distributed sensor network was developed that can measure noise levels and classify between 14 different acoustic scene and event classes based on audio signals recorded at different locations in a city

[69] . In this case, the sensor processing is performed in real time on the embedded platform Raspberry Pi 3. Previous research has explored novel methods for spectrogram data compression based on autoencoder networks

[71] . Recently, significant progress has been made through the use of methods from deep learning in the field of music signal processing (music information retrieval) in applications such as music transcription

[76] ,

[77] , chord detection

[78] , and instrument detection

[79] . In the field of industrial sound processing, new datasets have been established, and deep learning methods have been used, for example, to monitor the acoustic condition of electric motors

[75] .

[0016] The scenario addressed in this embodiment assumes several sound sources whose number and type are initially unknown and may change continuously. For source separation, several sound sources with similar characteristics, such as several loudspeakers, present a particularly large challenge

[80] .

[0017] To achieve high spatial resolution, several microphones must be used in the form of an array.

[72] In contrast to conventional audio recordings, which are mono (one channel) or stereo (two channels), such a recording scenario allows for accurate localization of sound sources around the listener.

[0018] Source separation algorithms typically leave artifacts, such as distortion and crosstalk between the sources [5], which can generally be perceived as disruptive by the listener. By remixing the tracks, such artifacts can be partially masked and therefore reduced

[10] .

[0019] To enhance "blind" source separation, additional information such as the detected number and type of sources or their estimated spatial location is often used (informed source separation

[74] ). For conferences with several active speakers, current analysis systems simultaneously estimate the number of speakers and their respective temporal activity. decision and then they can be separated by source separation

[66] .

[0020] At Fraunhofer IDMT, much work has been done in recent years on perceptually based evaluation of sound source separation algorithms

[73] .

[0021] In the field of music signal processing, real-time capable algorithms have been developed for separating isolated and accompanying instruments, utilizing the base frequency estimates of isolated instruments as additional information

[81] . An alternative approach for separating vocals from complex musical pieces based on deep learning methods has been proposed in

[82] . Specialized source separation algorithms have also been developed for applications in the context of industrial audio analysis [7].

[0022] Headphones have a significant influence on the acoustic perception of the surroundings. Depending on the headphone design, sound incident on the ear is attenuated to different degrees. In-ear headphones completely block the ear channel.

[85] Closed headphones that surround the pinna also strongly isolate the listener acoustically from the external environment. Open and semi-open headphones allow sound to pass through completely or partially.

[84] In many applications in everyday life, it is desirable for headphones to isolate unwanted ambient sounds more strongly than is possible with their design type.

[0023] Furthermore, active noise control (ANC) can attenuate the effects of external interference. This is achieved by recording the incident sound signals with the headphone microphone and then reproducing them through the loudspeaker so that these sound parts and the sound parts that penetrate the headphones cancel each other out through interference. Overall, this can achieve a strong acoustic isolation from the surroundings. However, in many everyday situations, this is associated with danger, so it would be desirable to be able to intelligently turn on this function on demand.

[0024] The first product allows microphone signals to be passed through headphones to reduce passive isolation. Therefore, in addition to prototypes

[86] , commercial products already exist that advertise the "Transparent Hearing" feature. For example, Sennheiser offers the feature with its AMBEO headset

[88] , and Bragi offers it with its "Dash Pro" product. However, this possibility is only the beginning. In the future, this functionality should be significantly expanded to allow individual signal portions (e.g., only audio signals or alarm signals) to be made exclusively audible on demand, in addition to turning ambient sound completely on and off. The French company Orosound allows the wearer of a "chilled earphone" headset to adapt the strength of the ANC using a slider

[89] . In addition, the voice of the conversation partner can also be induced during activated ANC. However, this only works if the conversation partner is positioned facing you within a 60° cone. Direction-independent adaptation is not possible.

[0025] US Patent Application Publication No. 2015195641 (see

[91] ) discloses a method implemented to generate an auditory environment for a user, the method comprising receiving a signal representative of a user's ambient auditory environment and processing the signal using a microprocessor to identify at least one sound type among a plurality of sound types in the ambient auditory environment. The method further comprises receiving user preferences for each of the plurality of sound types, modifying the signal for each sound type in the ambient auditory environment, and outputting the modified signal to at least one loudspeaker to generate the user's auditory environment.

[0026] Headphone equalization and room adaptation (or spatial / spatial adaptation or spatial / spatial compensation) for binaural reproduction in augmented reality (AR) are important issues.

[0027] In a typical scenario, a human listener wears acoustically (partially) transparent headphones through which he hears ambient sounds. Furthermore, an additional sound source is reproduced through the headphones, said sound source being embedded in the real surroundings in such a way that the listener cannot distinguish between the real acoustic scene and the additional sound.

[0028] Typically, the direction of head rotation and the listener's position in the room (or space) are tracked (6 degrees of freedom (6DoF)). decision Research has shown that better results (i.e., externalization and accurate localization) are achieved when the room acoustics of the recording and playback rooms match, or when the recording is adapted to the playback room.

[0029] In this case, an exemplary solution can be implemented as follows:

[0030] In a first step, measurements of the BRIR are made without headphones, either in an individualized manner or using an artificial head with a probe microphone.

[0031] In the second step, an analysis of the room characteristics of the recording chamber is performed based on the measured BRIR.

[0032] In the third step, measurements of the headphone transfer function are made using an artificial head, either in an individualized manner or by means of a co-located probe microphone, so that the equalization function can be calculated. decision will be done.

[0033] Optionally, in a fourth step, measurements of the room characteristics of the reproduction room, analysis of the acoustic characteristics of the reproduction room, and adaptation of the BRIR to the reproduction room may be performed.

[0034] Then, in a further step, a convolution (or folding) of the source, augmented with a correctly positioned and optionally adapted BRIR, is performed to obtain two raw channels: a convolution of the raw channels with an equalization function to obtain a headphone signal;

[0035] Finally, in a further step, playback of the headphone signal is performed via the headphones.

[0036] However, wearing headphones has the problem that the influence of the pinna on the BRIR is eliminated, i.e. the BRIR is different from that without headphones. This results in a different natural sound source being heard than without headphones, but a virtual augmented sound source being reproduced as if there were no headphones. [Prior art documents] [Patent documents]

[0037] [Patent Document 1] US Patent Application Publication No. 2015195641 Summary of the Invention [Problem to be solved by the invention]

[0038] Easy, fast and efficient room characterization of the reproduction room decision It is desirable to provide a concept that enables [Means for solving the problem]

[0039] Embodiments of the present invention are provided below.

[0040] Thus, claim 1 provides a system according to an embodiment of the invention, claim 19 provides a method and claim 20 provides a computer program.

[0041] A system according to an embodiment of the present invention generates a plurality of binaural room impulse responses. decision and a loudspeaker signal generator for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to a sound source signal of the at least one sound source. The analyzer analyzes the plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account an effect resulting from headphones being worn by a user. decision It is configured to:

[0042] Further provided is a method according to an embodiment of the present invention, the method comprising: Multiple binaural room impulse responses decision and generating at least two loudspeaker signals in response to a plurality of binaural room impulse responses and in response to a source signal of at least one sound source; Includes:

[0043] The plurality of binaural room impulse responses are configured such that each of the plurality of binaural room impulse responses takes into account the effect caused by a user wearing headphones. decision will be done.

[0044] Furthermore, there is provided a computer program according to an embodiment of the present invention having a program code for performing the method described above.

[0045] Next, preferred embodiments of the present invention will be described with reference to the drawings. [Brief explanation of the drawings]

[0046] [Figure 1] FIG. 1 illustrates a system according to one embodiment. [Figure 2] FIG. 1 illustrates a further system for assisting selective hearing according to a further embodiment. [Figure 3] FIG. 1 illustrates a further system for assisting selective hearing, further including a user interface. [Figure 4] FIG. 1 illustrates a system for assisting selective hearing that includes a hearing device with two corresponding loudspeakers. [Figure 5a] FIG. 1 illustrates a system for assisting selective hearing, including a housing structure and two loudspeakers. [Figure 5b] FIG. 1 illustrates a system for assisting selective hearing, including headphones with two loudspeakers. [Figure 6] FIG. 1 illustrates a system according to one embodiment that includes a remote device 190 that includes a detector and locator, as well as a sound type classifier, a signal portion modifier, and a signal generator. [Figure 7] FIG. 1 illustrates a system according to one embodiment, including five subsystems. [Figure 8] FIG. 1 illustrates a corresponding scenario according to one embodiment. [Figure 9] FIG. 1 illustrates a scenario according to an embodiment with four external sound sources. [Figure 10] FIG. 1 is a diagram showing a processing workflow for SH applications according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0047] FIG. 1 is a diagram illustrating a system according to one embodiment.

[0048] The system generates multiple binaural room impulse responses. decision The analyzer 152 includes an analyzer 152 for analyzing the

[0049] Furthermore, the system includes a loudspeaker signal generator 154 for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to the source signal of the at least one sound source.

[0050] The analyzer 152 analyzes the plurality of binaural room impulse responses so that each of the plurality of binaural room impulse responses takes into account the effect caused by the user wearing headphones. decision It is configured to:

[0051] In one embodiment, for example, the system may include headphones, which may be configured to output at least two loudspeaker signals.

[0052] According to one embodiment, for example, a headphone may include at least two headphone capsules and at least one microphone for measuring sound in each of the two headphone capsules, and for example, the at least one microphone for measuring sound may be disposed in each of the two headphone capsules. Here, for example, the analyzer 152 may analyze a plurality of binaural room impulse responses using measurements of the at least one microphone in each of the two headphone capsules. decision Headphones intended for binaural reproduction always comprise at least two headphone capsules (e.g. for different frequency ranges), and three or more capsules may also be provided.

[0053] In one embodiment, for example, at least one microphone in each of two headphone capsules generates one or more recordings of the sound situation in the reproduction room (or space) prior to playback of the at least two loudspeaker signals by the headphones, and calculates an estimate of the raw audio signal of at least one sound source from the one or more recordings. decision Then, the binaural room impulse responses of multiple binaural room impulse responses for the sound source in the playback room are calculated. decision The device may be configured to:

[0054] According to one embodiment, for example, at least one microphone of each of the two headphone capsules generates one or more further recordings of the sound situation in the reproduction room during playback of the at least two loudspeaker signals by the headphones, and subtracts the augmented signal from these one or more further recordings to obtain an estimate of the raw sound signal from the one or more sound sources. decision Then, the binaural room impulse responses of multiple binaural room impulse responses for the sound source in the playback room are calculated. decision The device may be configured to:

[0055] In one embodiment, for example, the analyzer 152 measures the acoustical characteristics of the reproduction room. decision and may be configured to adapt multiple binaural room impulse responses depending on the acoustic room characteristics.

[0056] According to one embodiment, for example, at least one microphone may be placed in each of the two headphone capsules to measure sounds near the entrance of the ear canal.

[0057] In one embodiment, for example, the system may include one or more additional microphones outside the two headphone capsules to measure the sound situation in the reproduction room.

[0058] According to one embodiment, for example, headphones headphone band For example, at least one of the one or more further microphones may include headphone band is placed on top.

[0059] In one embodiment, for example, the loudspeaker signal generator 154 may be configured to generate at least two loudspeaker signals by convolving each of a plurality of binaural room impulse responses with a source signal of a plurality of one or more source signals.

[0060] According to one embodiment, for example, the analyzer 152 may generate at least one of the plurality of binaural room impulse responses (or some or all of the binaural room impulse responses) in response to headphone movement. decision The device may be configured to:

[0061] In an embodiment, the system detects the movement of the headphones. decision For example, the sensor may be an accelerometer or other sensor with at least 3 DoF (three degrees of freedom) to capture head rotation. For example, a 6 DoF sensor (six degrees of freedom sensor) may be used.

[0062] Certain embodiments of the present invention address the technical challenge of auditory environments, where certain sounds are often very loud and distracting, making selective hearing desirable. While the human brain itself can perform selective hearing to some extent, intelligent technological assistants can significantly improve selective hearing. While hearing aids exist, just as eyeglasses help many people in modern life better perceive their environment, even people with normal hearing can benefit from assistance from intelligent systems in many situations. To realize an "intelligent hearable" (hearing device or hearing aid), a technological system must analyze the (acoustic) environment and identify individual sound sources so that they can be processed separately. While research into this issue has already been conducted, prior art has not yet achieved the analysis and processing of an entire acoustic environment in real time (transparent to the ear) and with high sound quality (content that sounds indistinguishable from the normal acoustic environment).

[0063] An improved concept for machine listening is provided below.

[0064] In the first step, measurements of the BRIR using headphones are performed either individually with a probe microphone or using headphones.

[0065] In the second step, an analysis of the room characteristics of the recording chamber is performed based on the measured BRIR.

[0066] Optionally, for example in a third step, before playback, at least one built-in microphone in each shell records the actual sound situation in the playback room. From these recordings, estimates of the raw sound signals of one or more sound sources are obtained. decision The BRIR of each sound source in the playback room is decision From this estimation, the acoustic characteristics of the playback room are decision The BRIR in the recording room is then adjusted accordingly.

[0067] Optionally, for example in a further step, during playback at least one built-in microphone in each shell records the actual sound situation in the playback room. From these recordings, the augmented signal is first subtracted and then an estimate of the raw sound signal of one or more sound sources is obtained. decision The BRIR of each sound source in the playback room is decision From this estimation, the acoustic characteristics of the playback room are decision and the BRIR of the playback room is adapted to it.

[0068] In a further step, a convolution of the source augmented with a correctly positioned and optionally adapted BRIR is performed to obtain the headphone signal.

[0069] Finally, in a further step, playback of the headphone signal is performed via the headphones.

[0070] In one embodiment, at least one microphone is placed within each headphone capsule, for example to measure sound near the entrance of the ear canal.

[0071] According to one embodiment, an additional microphone is optionally located outside the headphones, possibly to measure and analyze the sound situation in the reproduction room. headphone band It is also placed on the top side of

[0072] In an embodiment, identical natural and extended source sounds are realized.

[0073] The embodiment recognizes that measurements of headphone characteristics are not required.

[0074] Thus, the embodiments provide a concept for measuring the room characteristics of a reproduction room.

[0075] Some embodiments provide starting values ​​and (post)optimization for room adaptation. The provided concept also works when the room acoustics of the reproduction room change, for example when the listener moves to another room (or space).

[0076] Among other things, the embodiments are based on installing different technologies for assisting hearing in technical systems and then combining them in such a way that an improvement in sound and quality of life (e.g., desired sounds are louder, undesired sounds are softer, speech intelligibility is better) is achieved for people with normal hearing and for people with hearing loss.

[0077] FIG. 2 is a diagram illustrating a system for assisting selective hearing according to one embodiment.

[0078] The system includes a detector 110 for detecting source signal portions of one or more sound sources by using at least two received microphone signals of the auditory environment (or listening environment).

[0079] Additionally, the system includes a location manager for assigning location information to each of the one or more sound sources. decision Includes a container 120.

[0080] Additionally, the system includes an audio type classifier 130 for assigning an audio signal type to the audio source signal portion of each of the one or more audio sources.

[0081] Further, the system includes a signal portion modifier 140 for modifying the source signal portion of at least one of the one or more sound sources according to an audio signal type of the source signal portion of the at least one sound source to obtain a modified audio signal portion of the at least one sound source.

[0082] The analyzer 152 and the loudspeaker signal generator 154 of FIG.

[0083] The analyzer 152 of the signal generator 150 is configured to generate a plurality of binaural room impulse responses, the plurality of binaural room impulse responses being for each of the one or more sound sources depending on the position information of the sound source and the orientation of the user's head.

[0084] The loudspeaker signal generator 154 of the signal generator 150 is configured to generate at least two loudspeaker signals in response to a plurality of binaural room impulse responses and in response to the modified audio signal portion of the at least one sound source.

[0085] According to one embodiment, for example, the detector 110 may be configured to detect source signal portions of one or more sound sources by using a deep learning model.

[0086] In one embodiment, for example, the position decision The device 120 may generate position information for each of the one or more sound sources in response to the captured image or recorded video. decision The device may be configured to:

[0087] According to one embodiment, for example, the position decision The device 120 detects lip movements of a person in the video, and assigns position information to a sound source signal portion of one of the one or more sound sources according to the lip movements, thereby generating position information according to the video for each of the one or more sound sources. decision The device may be configured to:

[0088] In one embodiment, for example, the detector 110 determines one or more acoustic characteristics of the auditory environment in response to at least two received microphone signals. decision The device may be configured to:

[0089] According to one embodiment, for example, the signal generator 150 generates multiple binaural room impulse responses in response to one or more acoustic characteristics of the auditory environment. decision The device may be configured to:

[0090] In one embodiment, for example, the signal portion modifier 140 may be configured to select at least one sound source whose sound source signal portion is to be modified in response to a previously learned user scenario and modify it in response to the previously learned user scenario.

[0091] According to one embodiment, for example, the system may include a user interface 160 for selecting a previously learned user scenario from a group of two or more previously learned user scenarios. Figure 3 illustrates such a system according to one embodiment, further including such a user interface 160.

[0092] In one embodiment, for example, the detector 110 and / or the position decision The decoder 120 and / or the audio type classifier 130 and / or the signal modifier 140 and / or the signal generator 150 may be configured to perform parallel signal processing using a Hough transform, or using multiple VLSI chips, or by using multiple memristors.

[0093] According to one embodiment, for example, the system may include a hearing device 170 that functions as a hearing aid for a user with limited hearing capabilities and / or hearing impairment, the hearing device including at least two loudspeakers 171, 172 for outputting at least two loudspeaker signals. Figure 4 shows such a system according to one embodiment, including such a hearing device 170 with two corresponding loudspeakers 171, 172.

[0094] In one embodiment, for example, the system may include at least two loudspeakers 181, 182 for outputting at least two loudspeaker signals, and a housing structure 183 for accommodating the at least two loudspeakers, where at least one housing structure 183 is suitable for being fixed to a user's head 185 or any other body part of the user. Figure 5a shows such a housing structure 183 and a corresponding system including two loudspeakers 181, 182.

[0095] According to one embodiment, for example, the system may include headphones 180 including at least two loudspeakers 181, 182 for outputting at least two loudspeaker signals. Figure 5b shows corresponding headphones 180 with two loudspeakers 181, 182 according to one embodiment.

[0096] In one embodiment, for example, the detector 110 and the position decision The filter 120 and the sound type classifier 130 as well as the signal portion modifier 140 and the signal generator 150 may be integrated into a headphone 180 .

[0097] According to one embodiment shown in FIG. 6, for example, the system includes a detector 110 and a position decision The remote device 190 may include the remote device 190, which includes the audio classifier 120, the audio type classifier 130, the signal portion modifier 140, and the signal generator 150. In this case, for example, the remote device 190 may be spatially separated from the headphones 180.

[0098] In one embodiment, for example, remote device 190 may be a smartphone.

[0099] Embodiments do not necessarily use microprocessors, but rather use parallel signal processing steps such as Hough transforms, VLSI chips, or memristors for energy-efficient implementations, especially for artificial neural networks.

[0100] In an embodiment, the auditory environment is captured and reproduced spatially, using two or more signals for the representation of the input signal on the one hand, and also using spatial reproduction on the other hand.

[0101] In an embodiment, signal separation is performed by a deep learning (DL) model (e.g., CNN, RCNN, LSTM, Siamese network) that simultaneously processes information from at least two microphone channels, with at least one microphone in each hearable. According to the invention, several output signals (corresponding to individual sound sources) are separated by cross-analysis together with their respective spatial locations. decision If a recording means (microphone) is connected to the head, the position of the object changes as the head moves, which makes it possible to naturally focus on important / unimportant sounds, for example, by facing the sound object.

[0102] In some embodiments, the algorithm for signal analysis is based on, for example, a deep learning architecture. Alternatively, it uses variations by the analysis unit or variations by a separate network for aspect localization, detection, and sound separation. An alternative use of generalized cross-correlation (correlation vs. time offset) accounts for frequency-dependent shadowing / separation by the head, improving localization, detection, and sound source separation.

[0103] According to one embodiment, different source categories (e.g., speech, vehicle, male / female / voice of a child, warning tones, etc.) are learned by the detector in a training phase, where the source separation network is also trained on the localization network with target stimuli for high signal quality, as well as high accuracy of localization.

[0104] For example, the training steps described above use multi-channel speech data, and a first training round is typically performed in a laboratory using simulated or recorded speech data, followed by training runs in different natural environments (e.g., living rooms, classrooms, train stations, (industrial) production environments, etc.), i.e., transfer learning and domain adaptation.

[0105] Alternatively or additionally, the position detector may also detect the visual position of the sound source / sound source. decision In the case of speech, lip movements are correlated with the audio signal coming from the sound source separator to achieve more accurate localization.

[0106] After training, there is a DL model with the network architecture and associated parameters.

[0107] In some embodiments, the auralization is performed by binaural synthesis, which offers the additional advantage that, although it is not possible to completely remove undesired components, it is possible to reduce them to a level that is perceptible but not disturbing. This has the additional advantage of perceiving unexpected additional sources (warning signals, speech bubbles, ...) that would be missed if they were completely turned off.

[0108] According to some embodiments, analysis of the auditory environment is used not only to separate objects but also to analyze acoustic characteristics (e.g., reverberation time, initial time gap). These characteristics are then used in binaural synthesis to adapt pre-stored (possibly personalized) binaural room impulse responses (BRIRs) to the actual room (or space). By reducing room divergence, listeners have significantly reduced listening effort when understanding the optimized signal. Minimizing room divergence affects the externalization of auditory events and therefore the authenticity of spatial audio reproduction in a monitoring room. There are no known solutions in the prior art for speech understanding or general understanding of optimized signals.

[0109] In an embodiment, a user interface is used to determine which sound source is selected. decision According to the present invention, this is done by pre-learning different user scenarios such as "amplify voice directly in front of me" (single person conversation), "amplify voice in ±60 degree range" (group conversation), "suppress music and amplify music" (don't want to hear concert-goers), "silence everything" (want to be left alone), "suppress all voices and warning tones", etc.

[0110] Some embodiments are agnostic to the hardware used, i.e., open and closed headphones can be used. Signal processing can be built into the headphones, an external device, or the smartphone. Optionally, in addition to playing acoustically recorded and processed signals, signals can be played directly from the smartphone (e.g., music, phone calls).

[0111] In another embodiment, an ecosystem for "AI-assisted selective listening" is provided. The embodiment refers to "personalized auditory reality" (PARty). In such a personalized environment, the listener can amplify, reduce or modify defined acoustic objects. A series of analysis and synthesis processes should be performed to create a sound experience adapted to individual requirements. Research into targeted transformation stages forms an essential component for this.

[0112] Some embodiments provide analysis of the real sound environment and detection of individual acoustic objects, isolation, tracking and editability of available objects, and reconstruction and playback of modified acoustic scenes.

[0113] In an embodiment, sound event detection, sound event isolation, and suppression of some sound events are provided.

[0114] In an embodiment, AI methods (particularly deep learning-based methods) are used.

[0115] Embodiments of the present invention contribute to the development of technologies for recording, signal processing, and playback of spatial audio.

[0116] For example, embodiments create spatiality and three-dimensionality in a multimedia system with interacting users.

[0117] In this case, the embodiment is based on research knowledge of the perceptual and cognitive processing of spatial hearing / listening.

[0118] Some embodiments use two or more of the following concepts.

[0119] Scene decomposition: This involves spatial acoustic detection of the real environment, as well as parameter estimation and / or position-dependent sound field analysis.

[0120] Scene representation: This involves the representation and identification and / or efficient representation and storage of objects and / or environments.

[0121] Scene combination and reproduction: This involves the adaptation and transformation of objects and environments, and / or the rendering and auralization.

[0122] Quality assessment: This includes technical and / or auditory quality measurements.

[0123] Microphone positioning: This involves the application of a microphone array and appropriate audio signal processing.

[0124] Signal conditioning: This includes feature extraction as well as dataset generation for machine learning (ML).

[0125] Room and ambient acoustics estimation: This includes in-situ measurement and estimation of room acoustic parameters and / or providing room acoustic features for source separation and ML.

[0126] Auralization: This includes spatial audio reproduction with auditory adaptation to the environment and / or validation and evaluation and / or functional proof and quality estimation.

[0127] FIG. 8 illustrates a corresponding scenario according to one embodiment.

[0128] The embodiments combine concepts for sound source detection, classification, separation, localization, and enhancement, highlighting recent advances in each field and illustrating the connections between them.

[0129] In the following, we provide a consistent concept that can combine / detect / classify / localize and separate / enhance sound sources to provide the flexibility and robustness required for real-world SH.

[0130] Furthermore, embodiments provide a low latency concept suitable for real-time performance when dealing with the dynamics of real auditory scenes.

[0131] Some embodiments use concepts from deep learning, machine hearing, and smart headphones (smart hearables) to enable the listener to selectively modify the auditory scene.

[0132] Embodiments offer listeners the possibility to selectively enhance, attenuate, suppress or modify sound sources within an auditory scene using hearing devices such as headphones, earphones or the like.

[0133] FIG. 9 illustrates a scenario according to one embodiment with four external sound sources.

[0134] In Figure 9, the user is the center of the auditory scene. In this case, four external sound sources (S1-S4) are active around the user. The user interface allows the listener to influence the auditory scene. Sources S1-S4 can be attenuated, enhanced, or suppressed by their corresponding sliders. As can be seen in Figure 2, the listener can define sound sources or sound events that should be kept in the auditory scene or suppressed from it. In Figure 2, urban background noise should be suppressed, but alarms or phone calls should be preserved. At all times, the user has the possibility to play (or play) additional audio streams, such as music or radio, through the hearing device.

[0135] The user is typically the center of the system and controls the auditory scene by means of a control unit. The user can modify the auditory scene using a user interface such as that shown in Figure 9 or using any type of interaction, such as voice control, gestures, or gaze direction. Once the user provides feedback to the system, the next step consists of the detection / classification / location stage. In some cases, only detection is necessary, for example, if the user wants to preserve any speech occurring in the auditory scene. In other cases, classification may be necessary, for example, if the user wants to preserve fire alarms in the auditory scene, but not phone ringing or office noise. In some cases, only the location of the source is relevant to the system. This is, for example, the case with the four sound sources in Figure 9, where the user may want to remove or attenuate sound sources coming from a particular direction, regardless of the type or characteristics of the source. decision It is possible.

[0136] FIG. 10 shows a processing workflow for an SH application according to an embodiment.

[0137] First, the separation enhancement stage in Figure 10 modifies the auditory scene. This is done by suppressing, attenuating, or enhancing specific sound sources (e.g., or specific sound sources). As shown in Figure 10, an additional processing option in SH is noise control, which has the goal of removing or minimizing background noise in the auditory scene. Perhaps the most common and widespread technique for noise control is active noise control (ANC)

[11] .

[0138] Selective hearing is distinguished from virtual and augmented auditory environments by restricting selective hearing to applications where only real sound sources are modified in the auditory scene, without attempting to add virtual sound sources to the scene.

[0139] From a machine hearing perspective, selective hearing applications require techniques to automatically detect, localize, classify, separate, and enhance sound sources. To further clarify the terminology related to selective hearing, the following terms are defined and their differences and relationships are highlighted:

[0140] In embodiments, for example, sound source localization is used, which refers to the ability to detect the position of a sound source within an auditory scene. In the context of audio processing, sound source location typically refers to the direction of arrival (DOA) of a given sound source, which can be given as either a 2D coordinate (azimuth) or a 3D coordinate if it includes elevation. Some systems also estimate the distance from the sound source to the microphone as position information [3]. In the context of music processing, location often refers to the panning of the sound source in the final mix, which is typically given as an angle in degrees [4].

[0141] According to an embodiment, for example, it is possible to determine whether any instances of a given sound source type are present in an auditory scene. decision An example of a detection task is determining whether a speaker is present in a scene. decision In this context, the number of speakers in a scene or the identity of the speakers can be recorded. decision This is outside the scope of sound source detection, which can be understood as a binary classification task where the classes correspond to "source present" and "source absent".

[0142] In embodiments, for example, sound source classification is used to assign a class label from a set of predetermined classes to a given sound source or a given sound event. An example of a classification task is determining whether a given sound source corresponds to speech, music, or ambient noise. decision The goal is to detect the presence or absence of a sound source. Sound source classification and detection are closely related concepts. In some cases, classification systems include a detection stage by considering "no class" as one of the possible labels. In these cases, the system implicitly learns to detect the presence or absence of a sound source and is not forced to assign a class label if there is not enough evidence that one of the sources is active.

[0143] According to an embodiment, source separation is used to refer to the extraction of a given sound source from a sound mixture or auditory scene. An example of source separation is the extraction of a singing voice from a sound mixture where, in addition to the singer, other instruments are playing simultaneously [5]. Source separation becomes relevant in selective listening scenarios, as it allows for the suppression of sound sources that are not of interest to the listener. Some sound separation systems implicitly perform a detection task before extracting the sound source from the mixture. However, this is not necessarily the rule, and therefore, it is important to emphasize the distinction between these tasks. Furthermore, separation often serves as a preprocessing stage for other types of analysis, such as source enhancement [6] or classification [7].

[0144] In embodiments, for example, sound source identification is used, which goes one step further and aims to identify specific instances of sound sources within an audio signal. Speaker identification is perhaps the most common use of sound source identification today. In this task, the goal is to identify whether a particular speaker is present in a scene. In the example of FIG. 1, the user has selected "Speaker X" as one of the sound sources contained in the auditory scene. This requires technology that goes beyond sound detection and classification and requires speaker-specific models that enable this accurate identification.

[0145] In some embodiments, for example, source enhancement is used to refer to a process that increases the salience of a given sound source in an auditory scene [8]. In the case of audio signals, the goal is often to increase their perceptual quality and intelligibility. A common scenario for audio enhancement is the denoising of audio corrupted by noise [9]. In the context of music processing, source enhancement is related to the concept of remixing and is often performed to make one instrument (sound source) more prominent in the mix. Remixing applications often use audio separation front-ends to access individual sound sources and modify the characteristics of the mixture

[10] . While a source separation stage can occur before source enhancement, this is not always the case, and therefore we emphasize the distinction between these terms.

[0146] In the field of sound source detection, classification, and identification, for example, some embodiments use one of the following concepts, such as acoustic scene and event detection and classification

[18] . In this context, a method for audio event detection (AED) in a home environment has been proposed, where the goal is to detect the time boundary of a given sound event within a 10-second recording

[19] ,

[20] . In this particular case, 10 sound event classes were considered, including cat, dog, talking, alarm, and tap water. Methods for polyphonic sound event detection (several simultaneous events) have also been proposed in

[21] ,

[22] . In

[21] , a method for polyphonic sound event detection is proposed, where a total of 61 sound events from a real-world context are detected using a binary activity detector based on a bidirectional long short-term memory (BLSTM) recurrent neural network (RNN).

[0147] For example, to handle weakly labeled data, some embodiments incorporate a temporal attention mechanism to focus on specific regions of the signal for classification

[23] . The problem of noisy labels in classification is particularly relevant for selective hearing applications, where class labels are highly variable and high-quality annotation is prohibitively costly

[24] . Noisy labels in sound event classification tasks have been addressed in

[25] , which presents a noise-robust loss function based on categorical cross-entropy, as well as a method for evaluating both noisy and manually labeled data. Similarly,

[26] presents a system for sound event classification based on a convolutional neural network (CNN) that incorporates a noisy label validation step based on the CNN's predicted consensus over multiple segments of training examples.

[0148] For example, some embodiments provide simultaneous detection and localization of sound events. Thus, some embodiments perform detection as a multi-label classification task, such as in

[27] , where the location is given as the 3D coordinates of the direction of arrival (DOA) of each sound event.

[0149] Some embodiments use concepts of voice activity detection and speaker recognition / identification for speech recognition. Voice activity detection has been addressed in noisy environments using denoising autoencoders

[28] , recurrent neural networks

[29] , or as end-to-end systems using raw waveforms

[30] . For speaker recognition applications, numerous systems have been proposed in the literature

[31] , most of which focus on increasing robustness to different conditions, for example, using data augmentation or improved embeddings that facilitate recognition

[32] -

[34] . Therefore, some embodiments use these concepts.

[0150] Further embodiments use concepts for instrument classification for sound event detection. Instrument classification in both monophonic and polyphonic settings has been addressed in

[35] and

[36] . In

[35] , dominant instruments in 3-second audio segments are classified among 11 instrument classes, and several aggregation techniques are proposed. Similarly,

[37] proposes a method for instrument activity detection that can detect instruments at a finer time resolution of 1 second. A considerable amount of research has been conducted in the field of singing voice analysis. In particular, methods such as

[38] have been proposed for the task of detecting segments in recordings where singing is active. Some embodiments use these concepts.

[0151] Some of the embodiments use one of the concepts described below for sound source localization. Sound source localization is closely related to the problem of sound source counting, since the number of sound sources in an auditory scene is usually unknown in real-world applications. Some systems operate under the assumption that the number of sources in a scene is known. This is the case, for example, of the model presented in

[39] , which uses a histogram of active intensity vectors to localize sources. From a supervised perspective,

[40] proposes a CNN-based algorithm for estimating the DOAs of multiple speakers in an auditory scene using a phase map as an input representation. In contrast, some works in the literature jointly estimate the number of sources in a scene and their location information. This is the case in

[41] , where a system for multi-speaker localization in noisy and reverberant environments is proposed. The system uses a complex-valued Gaussian Mixture Model (GMM) to estimate both the number of sources and their localization. The concepts described there are used by some embodiments.

[0152] Sound source localization algorithms can be computationally demanding because they often involve scanning a large space around the auditory scene

[42] . To reduce the computational requirements in localization algorithms, some embodiments use concepts that reduce the search space by using clustering algorithms

[43] or by performing a multi-resolution search with established methods, such as methods based on the steered response power phase transform (SRP-PHAT)

[42] . Other methods impose sparsity constraints and assume that only one sound source dominates in a given time-frequency region

[44] . Recently, an end-to-end system for azimuth detection directly from raw waveforms was proposed in

[45] . Some embodiments use these concepts.

[0153] Some embodiments use concepts described below for sound source separation (SSS), especially from the field of speech and music separation.

[0154] In particular, some embodiments use the concept of speaker-independent separation: separation is performed in situ without any prior information about the speakers in the scene

[46] . Some embodiments also estimate the spatial location of the speakers to perform separation

[47] .

[0155] Given the importance of computational performance in selective hearing applications, research conducted with the specific goal of achieving low latency is particularly important. Several studies have proposed low-latency speech separation (<10 ms) with little available training data

[48] . To avoid the delays introduced by framing analysis in the frequency domain, some systems approach the separation problem by carefully designing filters applied in the time domain

[49] . Other systems achieve low-latency separation by directly modeling the time-domain signal using an encoder-decoder framework

[50] . In contrast, some systems attempt to reduce the framing delay in frequency-domain separation techniques

[51] . These concepts are adopted by several embodiments.

[0156] Some embodiments use concepts for music sound separation (MSS), such as those for reed instrument accompaniment separation

[52] , to extract musical sources from audio mixtures [5]. These algorithms attempt to obtain the most prominent audio source in a mixture, regardless of its class label, and separate it from the remaining accompaniments. Some embodiments use concepts for singing voice separation

[53] . In most cases, either specific source models

[54] or data-driven models

[55] are used to capture the characteristics of singing voices. While systems such as those proposed in

[55] do not explicitly incorporate a classification or detection stage to achieve separation, the data-driven nature of these approaches allows these systems to implicitly learn to detect singing voices with a certain accuracy before separation. Another class of algorithms in the music domain attempts to perform separation using only audio source locations, without attempting to classify or detect the audio sources before separation [4].

[0157] Some embodiments use concepts of active noise control (ANC), such as active noise cancellation (ANC). ANC systems primarily aim to eliminate background noise for headphone users by introducing and canceling noise-canceling signals

[11] . ANC can be considered a special case of SH and faces similarly stringent performance requirements

[14] . Some research has focused on active noise control in specific environments, such as automobile cabins

[56] or industrial scenarios

[57] . The work in

[56] analyzes the elimination of different types of noise, such as road noise and engine noise, and calls for a unified noise control system that can address different types of noise. Some research has focused on developing ANC systems to eliminate noise over specific spatial regions. In

[58] , ANC over spatial regions is addressed using spherical harmonics as basis functions to represent the noise field. Some embodiments use the concepts described herein.

[0158] Some of the embodiments use the concept for audio source extension.

[0159] In the context of speech enhancement, one of the most common applications is enhancing speech corrupted by noise. Much research has focused on phase processing for single-channel speech enhancement [8]. From the perspective of deep neural networks, the problem of speech denoising has been addressed in end-to-end systems using denoising autoencoders in

[59] , nonlinear regression problems between clean and noisy speech using deep neural networks (DNNs) in

[60] , and generative adversarial networks (GANs) in

[61] . Often, speech enhancement is applied as a front-end to automatic speech recognition (ASR) systems, as in

[62] , which approaches speech enhancement with an LSTM RNN. Speech enhancement is also often performed in conjunction with source separation techniques, which may first extract speech and then apply enhancement techniques to the separated speech signal [6]. The concepts described herein are used by some embodiments.

[0160] In most cases, source enhancement in the context of music refers to applications for creating music remixes. In contrast to speech enhancement, which often assumes that speech is only impaired by noise sources, music applications almost always assume that other sound sources (musical instruments) are playing simultaneously with the source to be emphasized. For this reason, music remix applications are always preceded by a source separation stage. For example, in

[10] , early jazz recordings were remixed by applying lead accompaniment and harmonic percussion separation techniques to achieve a better tonal balance in the mixture. Similarly,

[63] investigated the use of different vocal separation algorithms to change the relative volume of the vocal voice and the backing track, showing that a 6 dB increase was possible by introducing slight but audible distortion into the final mixture. In

[64] , the authors investigated how to improve music perception for cochlear implant users by applying sound source separation techniques to achieve a new mix. The concepts described there are used by some embodiments.

[0161] One of the biggest challenges in selective hearing applications relates to the stringent processing time requirements. To maintain naturalness and perceived quality for the user, the complete processing workflow must be performed with minimal delay. The maximum acceptable latency of a system depends heavily on the application and the complexity of the auditory scene. For example, McPherson et al. proposed 10 ms as an acceptable latency criterion for interactive music interfaces

[12] . For networked music performance, the authors of

[13] reported that latency ranges from 20–25 ms to 50–60 ms. However, active noise control / cancellation (ANC) technologies require ultra-low latency processing for better performance. In these systems, the amount of acceptable latency depends on both frequency and attenuation, but can be as low as 1 ms for approximately 5 dB attenuation of frequencies below 200 Hz

[14] . A final consideration in SH applications refers to the perceived quality of the modified auditory scene. A considerable amount of work has been devoted to methodologies for reliable assessment of speech quality in various applications

[15] ,

[16] ,

[17] . However, the challenge of SH is managing the clear trade-off between processing complexity and perceptual quality. Some embodiments use the concepts described therein.

[0162] Some embodiments use concepts for counting / calculating and localizing as described in

[41] , localizing and detecting as described in

[27] , separating and classifying as described in

[65] , and separating and counting as described in

[66] .

[0163] Some embodiments use concepts to increase the robustness of current machine listening methods, as described in

[25] ,

[26] ,

[32] ,

[34] , and new emerging directions include domain adaptation

[67] and training on datasets recorded on multiple devices

[68] .

[0164] Some embodiments can handle raw waveforms and use concepts to improve the computational efficiency of machine listening methods, such as those described in

[48] , or concepts described in

[30] ,

[45] ,

[50] ,

[61] .

[0165] Some embodiments implement an integrated optimization scheme that detects / classifies / localizes and separates / enhances in a combined manner so that sound sources within a scene can be selectively modified, while the independent detection, separation, localization, classification, and enhancement methods are reliable and provide the robustness and flexibility required for SH.

[0166] Some embodiments are suitable for real-time processing with a good trade-off between algorithm complexity and performance.

[0167] Some embodiments combine ANC with machine hearing, for example, an auditory scene is first classified and then ANC is selectively applied.

[0168] Further embodiments are provided below.

[0169] In order to augment a real auditory environment with virtual sound objects, the transfer functions from each location of the sound object to each location of the listener in the room must be well known.

[0170] The transfer function maps the characteristics of the sound source, the direct sound between the object and the user, and all reflections that occur in the room. To ensure correct spatial audio reproduction for the actual room acoustics that the listener is currently in, the transfer function must also map the room acoustics within the listener's room with sufficient accuracy.

[0171] In audio systems suited to representing individual audio objects at different locations in a room, the challenge is to properly detect and separate the individual audio objects when multiple audio objects are present, and the audio signals for each object overlap at the recording or listening position in the room. The room acoustics and audio signal overlap change as the objects and / or listening positions in the room change.

[0172] Due to relative motion, the estimation of room acoustic parameters must be performed sufficiently quickly. Here, low estimation latency is more important than high accuracy. High accuracy is required when the positions of the sound source and receiver do not change (static case). In the proposed system, the room acoustic parameters, as well as the room geometry and listener positions, are estimated or extracted from a stream of audio signals. The audio signals are recorded in a real environment where the sound source(s) and receiver(s) can move in any direction and the sound source(s) and / or receiver(s) can change their orientation arbitrarily.

[0173] The audio signal stream may be the result of any microphone setup, including one or more microphones. The stream is fed to a signal processing stage for pre-processing and / or further analysis. The output is then fed to a feature extraction stage, which estimates room acoustic parameters, such as T60 (reverberation time), DRR (direct-to-reverberant ratio), etc.

[0174] The second data stream is generated by a 6DoF sensor ("six degrees of freedom": three dimensions of room position and gaze direction) that captures the orientation and position of the microphone setup. The position data stream is fed into a 6DoF signal processing stage for pre-processing or further analysis.

[0175] The output of the 6DoF signal processing, audio feature extraction stage, and preprocessed microphone streams are fed into a machine learning block, which estimates the auditory space, i.e., the listening room (size, geometry, reflective surfaces), and the location of the microphone field within the room. Furthermore, to enable more robust estimation, a user behavior model is applied. This model takes into account constraints on human motion (e.g., continuous movement, speed, etc.) as well as probability distributions for various types of motion.

[0176] Some embodiments achieve blind estimation of room acoustic parameters using arbitrary microphone placements, by adding user position and pose information, and by analyzing the data using machine learning methods.

[0177] For example, systems according to embodiments can be used for acoustically augmented reality (AAR), where a virtual room impulse response must be synthesized from estimated parameters.

[0178] Some embodiments include removing reverberation from recorded signals. An example of such an embodiment is a hearing aid for people with normal hearing and for people with hearing impairments. In this case, reverberation can be removed from the input signal of a microphone setup with the help of estimated parameters.

[0179] A further application is the spatial synthesis of sound scenes generated in rooms other than the current auditory space. For this purpose, the room acoustics parameters that are part of the sound scene are adapted to the room acoustics parameters of the auditory space.

[0180] In the case of binaural synthesis, for this purpose the available BRIRs are adapted to the different acoustic parameters of the auditory space.

[0181] In one embodiment, one or more room acoustics parameters are decision An apparatus for performing the method is provided.

[0182] The device is configured to acquire microphone data comprising one or more microphone signals.

[0183] Additionally, the device is configured to obtain tracking data regarding the position and / or orientation of the user.

[0184] Additionally, the device may calculate one or more room acoustics parameters in response to the microphone data and the tracking data. decision It is configured to:

[0185] According to one embodiment, for example, the device may calculate one or more room acoustics parameters in response to the microphone data and the tracking data. decision The device may be configured to use machine learning to perform the task.

[0186] In an embodiment, for example, the device may be configured to use machine learning, in that the device may be configured to use neural networks.

[0187] According to one embodiment, for example, the device may be configured to use cloud-based processing for machine learning.

[0188] In one embodiment, for example, the one or more room acoustics parameters may include reverberation time.

[0189] According to one embodiment, for example, the one or more room acoustics parameters may include a directivity to reverberation ratio.

[0190] In one embodiment, for example, the tracking data may include x, y, and z coordinates to label the user's location.

[0191] According to an embodiment, for example, the tracking data may include pitch, yaw, and roll coordinates to label the user's orientation.

[0192] In an embodiment, for example, the device may be configured to transform one or more microphone signals from the time domain to the frequency domain, for example, the device may be configured to extract one or more features of the one or more microphone signals in the frequency domain, and the device may calculate one or more room acoustic parameters according to the one or more features. decision The device may be configured to:

[0193] According to one embodiment, for example, the device may be configured to use cloud-based processing to extract one or more features.

[0194] In one embodiment, for example, the device may include a microphone arrangement of several microphones for recording several microphone signals.

[0195] According to one embodiment, for example, the microphone arrangement may be configured to be worn on the body of a user.

[0196] In an embodiment, for example, the system described above may include: decision The device may further include the above-described device for:

[0197] According to one embodiment, for example, the signal portion modifier 140 and / or the signal generator 150 may be configured to perform the generation of at least one of the plurality of binaural room impulse responses for each of the one or more sound sources depending on said at least one of the one or more room acoustics parameters.

[0198] FIG. 7 shows a system according to one embodiment that includes five subsystems (Subsystems 1-5).

[0199] Subsystem 1 may include a microphone setup of one, two, or more individual microphones, which may be combined into a microphone field if one or more microphones are available. The microphones may be positioned and arranged relative to one another in any way. The microphone configuration may be part of a device worn by the user or may be a separate device placed in the room of interest.

[0200] Subsystem 1 also includes a tracking device that measures the user's translational position in the room and the user's head pose, up to 6DoF (x coordinate, y coordinate, z coordinate, pitch angle, yaw angle, roll angle).

[0201] The tracking device may be placed on the user's head or may be split into several sub-devices to measure the required DoF, and may or may not be placed on the user.

[0202] Thus, subsystem 1 represents an input interface including a microphone signal input interface 101 and a position information input interface 102 .

[0203] Subsystem 2 includes signal processing of the recorded microphone signals. This includes frequency conversion and / or time-domain based processing. Furthermore, it includes methods for combining different microphone signals to achieve field processing. Feedback from System 4 is possible to adapt the parameters of the signal processing in Subsystem 2. The signal processing block for the microphone signals may be part of the device in which the microphones are integrated or may be part of a separate device. It may also be part of cloud-based processing.

[0204] Subsystem 2 also includes signal processing for the recorded tracking data. This can include frequency transform and / or time-domain based processing. It also includes methods for improving the technical quality of the signal by using noise suppression, smoothing, interpolation, and extrapolation. It also includes methods for deriving higher-level information, including speed, acceleration, path direction, idle time, range of motion, and path of motion. It also includes predictions of near-future path of motion and near-future speed. The signal processing block for the tracking signal can be part of the tracking device, a separate device, or part of cloud-based processing.

[0205] Subsystem 3 includes the extraction of processed microphone features.

[0206] The feature extraction block may be part of the user's wearable device, a separate device, or part of cloud-based processing.

[0207] Subsystems 2 and 3, together with their modules 111 and 121, implement, for example, detector 110, speech type classifier 130, and signal portion modifier 140. For example, subsystem 3, module 121 can output the results of speech classification to subsystem 2, module 111 (feedback). For example, subsystem 2, module 112 can output the results of speech classification to subsystem 2, module 111 (feedback). decision Additionally, in one embodiment, subsystems 2 and 3 may also implement signal generator 150, for example, by subsystem 2, module 111 generating binaural room impulse responses and loudspeaker signals.

[0208] Subsystem 4 contains methods and algorithms for estimating room acoustic parameters using the processed microphone signals, extracted microphone signal features, and processed tracking data. The output of this block is the room acoustic characteristic parameters as idle data, as well as control and variation of parameters for microphone signal processing in subsystem 2. Machine learning block 131 may be part of the user's device, a separate device, or part of cloud-based processing.

[0209] Additionally, Subsystem 4 includes post-processing of room acoustic idle data parameters (e.g., in block 132). This includes outlier detection, combining individual parameters into new parameters, smoothing, extrapolation, interpolation, and validation. This block also obtains information from Subsystem 2, including the user's near-future position in the room, to estimate near-future acoustic parameters. This block may be part of the user's device or a separate device, or it may be part of cloud-based processing.

[0210] Subsystem 5 includes storage and allocation of room acoustics parameters for downstream systems (e.g., in memory 141). Parameter allocation may be accomplished just-in-time and / or time-responsively stored. Storage may be performed in a device located on or near the user, or in a cloud-based system.

[0211] Use cases for embodiments of the present invention are described below.

[0212] One embodiment use case is home entertainment, involving a user in a home environment. For example, a user may want to focus on a particular playback device, such as a TV, radio, PC, or tablet, and suppress other sources of disturbance (other users' or children's devices, construction noise, street noise). In this case, the user positions themselves near the preferred playback device and selects the device or its location. Regardless of the user's location, the selected device or sound source location is acoustically emphasized until the user cancels their selection.

[0213] For example, a user moves closer to a target sound source, the user selects the target sound source via an appropriate interface, and the hearable accordingly adapts the audio playback based on the user's position, the user's gaze direction, and the target sound source, ensuring that the target sound source is well understood even in the presence of noise interference.

[0214] Alternatively, the user moves near a particularly interfering sound source, the user selects this interfering sound source via an appropriate interface, and the hearable adjusts the audio playback accordingly based on the user's position, the user's gaze direction, and the interfering sound source, explicitly adjusting for the interfering sound source.

[0215] A further use case for a further embodiment is a cocktail party where the user is positioned between several speakers.

[0216] In the presence of many speakers, for example, the user may wish to focus on one (or more) of them and tune out or attenuate other sources of disturbance. In this use case, control of the hearable should require little interaction from the user. Control of the strength of selectivity based on biosignals or detectable indicators of speech difficulty (frequent questions, foreign words, strong diamond words) would be optional.

[0217] For example, speakers may be randomly distributed and move relative to the listener. Additionally, there may be periodic pauses in the audio as new speakers are added or others leave the scene. Perhaps external disturbances, such as music, are relatively loud. Selected speakers may be acoustically emphasized and become recognizable after pauses in speech, changes in their position or posture, etc.

[0218] For example, the hearable recognizes speakers in the user's vicinity. With appropriate controllability (e.g., gaze direction, attention control), the user can select a preferred speaker. By adapting the audio playback depending on the user's gaze direction and the selected target sound source, the hearable can fully understand the target sound source even in the presence of noise interference.

[0219] Alternatively, if a user is directly addressed by a (previously) non-preferred speaker, the user must at least be audible to ensure natural communication.

[0220] Another use case of another embodiment is in an automobile where the user is located in his / her own (or her) car. While driving, the user wants to actively direct his / her acoustic attention to specific playback devices, such as a navigation device, a radio, or a conversation partner, so that he / she can better understand them next to interfering noises (wind, motor, passengers).

[0221] For example, the user and target sound source are located at fixed positions in a car. The user is stationary relative to the reference system, but the vehicle itself is moving. This requires an adaptive tracking solution. The selected sound source location is acoustically emphasized until the user cancels the selection or a warning signal stops the device functioning.

[0222] For example, a user is in a car and the surroundings are detected by the device. With appropriate controllability (e.g., speed awareness), the user can switch between target sound sources, and the hearable adapts the audio playback according to the user's gaze direction and the selected target sound source so that the target sound source can be fully understood even in the presence of noise interference.

[0223] Alternatively, for example, a traffic-related warning signal may interrupt the normal flow and cancel the user's selection, after which resumption of normal flow may be implemented.

[0224] Another use case for further embodiments is live music and relates to guests at live music events. For example, guests at a concert or live music performance may wish to increase their concentration on the performance with the help of someone who can hear them and wish to ignore other guests who behave in a disruptive manner. Furthermore, the audio signal itself may be optimized, for example to balance out unfavorable listening positions or room acoustics.

[0225] For example, the user is located among many sources of disturbance. However, in most cases, performance is relatively large. The target sound source is located in a fixed position or at least in a predetermined area, while the user is highly mobile (e.g., the user may be dancing). The selected sound source location is acoustically emphasized until the user cancels the selection or until a warning signal stops the device functioning.

[0226] For example, the user selects a stage area or a musician as the target sound source. With appropriate controllability, the user can define the stage / musician position and the hearable will adapt the sound reproduction according to the user's gaze direction and the selected target sound source so that the target sound source can be fully understood even in the presence of noise interference.

[0227] Alternatively, for example, warning information (e.g., evacuation in case of an outdoor event, approaching thunderstorm) and warning signals can interrupt the normal flow and cancel the user's selection, after which there is a resumption of normal flow.

[0228] A further use case of another embodiment is major events and relates to guests at major events, so hearables can be used at major events (e.g., football stadiums, ice hockey stadiums, large concert halls, etc.) to highlight the voices of family and friends that would otherwise be drowned out by crowd noise.

[0229] For example, a large event with many attendees takes place in a stadium or large concert hall. A group (family, friends, school class) attends the event and is located outside or inside the event location where a large crowd is walking around. One or more children lose eye contact with the group and seek out the group despite the high noise level caused by the noise. The user then turns off voice recognition and the hearable no longer amplifies the sound.

[0230] For example, a person in the group can select the lost child's voice on the hearable, the hearable will find the voice, the hearable will amplify the voice, and the user can recover the lost child (faster) based on the amplified voice.

[0231] Alternatively, for example, a missing child may also wear a hearable and select a parent's voice. The hearable amplifies the parent's voice. The amplification allows the child to locate their parents. This allows the child to walk back to their parent. Alternatively, for example, a missing child may also wear a hearable and select a parent's voice. The hearable finds the parent's voice(s) and announces the distance to the voice(s). This makes it easier for the child to find their parent. Optionally, playback of an artificial voice from the hearable may be provided to announce the distance.

[0232] For example, coupling of hearables for selective amplification of sound is provided and sound profiles are stored.

[0233] A further use case of a further embodiment is recreational sports and concerns recreational athletes. Listening to music while playing sports is popular; however, it also poses risks. Warning signals or other road users may not be heard. In addition to playing music, the hearables can respond to warning signals or voices and temporarily interrupt music playback. In this context, a further use case is sports in small groups. Hearables of a sports group can be connected to ensure good communication during sports while suppressing other disturbing noises.

[0234] For example, users are mobile and possible warning signals are overlapped by many sources of disturbance. The problem is that not all warning signals are potentially relevant to the user (distant sirens in the city, horns in the street). This causes the hearable to automatically stop music playback and acoustically emphasize the communication partner's warning signal until the user cancels the selection. The music then plays normally.

[0235] For example, a user is engaged in sports and listening to music through a hearable. An alert signal or voice relating to the user is automatically detected, and the hearable pauses the music playback. The hearable adapts the audio playback to fully understand the target sound source / acoustic environment. The hearable then continues playing the music automatically (e.g., after the alert signal has ended) or as requested by the user.

[0236] Or, for example, athletes in a group can connect their hearables, optimizing speech intelligibility between group members and suppressing other distracting noises.

[0237] Another use case of another embodiment is snoring suppression, which concerns all people who want to sleep when their snoring partner disturbs their rest at night and causes problems in their sleep. The hearable provides a sense of security, as it suppresses the snoring sound, guarantees a good night's rest and provides safety in the home. At the same time, the hearable lets other sounds through (baby crying, alarms, etc.) so that the user is not completely acoustically isolated from the outside world. For example, snoring detection is provided.

[0238] For example, a user has trouble sleeping due to snoring sounds, and by using the hearable, the user can sleep better again, which has a stress-reducing effect.

[0239] For example, a user wears a hearable while sleeping, he / she switches the hearable to a sleep mode that suppresses all snoring sounds, and after going to sleep, turns the hearable off again.

[0240] Alternatively, it can silence other sounds such as construction noise, lawn mower noise, etc. while you sleep.

[0241] A further use case of a further embodiment is as a diagnostic device for the user in everyday life. The hearable records preferences (e.g., which sound source is selected, which attenuation / amplification is selected) and creates a profile with trends over the period of use. This data may make it possible to draw conclusions about changes in hearing ability. The aim is to detect hearing loss as soon as possible.

[0242] For example, a user carries the device with them in their daily life or the mentioned use case for months or years, and the hearable generates analytics based on the selected settings and outputs warnings and recommendations to the user.

[0243] For example, a user wears a hearable for an extended period of time (months to years), the device generates an analysis based on hearing preferences, and the device outputs recommendations and warnings in case of the onset of hearing loss.

[0244] A further use case of another embodiment is as a therapeutic device, relating to users with hearing impairments in their daily lives. In its role as a transitional device on the way to a hearing device, potential patients are assisted as early as possible, thus treating dementia preventatively. Other possibilities are use as a concentration trainer (e.g., in the case of ADHS), treatment of tinnitus, and stress reduction.

[0245] For example, a listener may have a hearing problem or attention deficit and temporarily / interim use a hearable as a hearing device. Depending on the hearing problem, the hearing device may alleviate it, for example, by amplifying all signals (hardness of hearing), by high selectivity of preferred sound sources (attention deficit), or by playing therapeutic sounds (treatment of tinnitus).

[0246] The user, independently or based on a doctor's advice, selects the form of treatment and makes the desired adjustments, and the hearable carries out the selected treatment.

[0247] Alternatively, the hearable can detect hearing problems from the UC-PRO1 and automatically adapt playback based on the detected problem and notify the user.

[0248] A further use case of a further embodiment is public sector work and concerns public sector employees, who are subjected to high levels of noise at work (hospitals, pediatricians, airport counters, educators, restaurant industry, service counters, etc.) wear hearables to highlight and better communicate what one or a few people are saying, for example through stress reduction, and for better safety at work.

[0249] For example, employees experience high levels of noise in their work environment and need to talk to clients, patients, or coworkers despite the background noise without being able to switch to a quieter environment. Hospital employees experience high levels of noise from medical devices ringing and beeping (or any other work-related noise) and must still be able to communicate with patients or coworkers. Pediatricians and educators must work amidst the noise or screaming of children and be able to talk to parents. At airport counters, employees have difficulty understanding airline passengers when the noise level is high in the airport concourse. Waiters have difficulty hearing customers' orders amid the noise of a favorite restaurant. Then, for example, the user turns off the audio selection and the hearable no longer amplifies the audio.

[0250] For example, a person turns on an equipped hearable. The user sets the hearable to audio selection for nearby sounds, and the hearable amplifies the nearby sounds, or a small number of nearby sounds, while simultaneously suppressing background noise. In this case, the user can better understand the relevant sounds.

[0251] Alternatively, a person sets their hearable to continuous noise suppression. The user turns on the function to detect and amplify available sounds, so the user can continue working at a lower noise level. When directly addressed from within x meters, the hearable then amplifies the sound. Thus, the user can converse with others at a lower noise level. After the conversation, the hearable returns to noise suppression mode, and after work, the user turns the hearable off again.

[0252] Another use case of another embodiment is passenger transportation, and concerns users of automobiles for passenger transportation. For example, users and drivers of passenger transportation vehicles want to minimize passenger distractions while driving. Although passengers are a major source of distraction, communication with them is sometimes necessary.

[0253] For example, the user or driver and the disturbance source are located at fixed positions in a car. The user is stationary relative to the reference system, but the vehicle itself is moving. This requires an adaptive tracking solution. Therefore, passenger sounds and conversations are acoustically suppressed by default unless communication is taking place.

[0254] For example, the hearable will suppress noise for the occupants by default, and the user can manually override the suppression through appropriate control possibilities (voice recognition, buttons in the vehicle), where the hearable will adapt the audio playback according to the selection.

[0255] Alternatively, the hearable may detect when the passenger is actively speaking to the driver and temporarily suspend noise suppression.

[0256] Another use case of further embodiments is in schools and education, involving teachers and students in a class. In one example, the hearable has two roles and the device functions are partially combined. The teacher / speaker's device suppresses distracting noise and amplifies speech / questions from students. The listener's hearable may also be controlled via the teacher's device. Thus, particularly important content can be emphasized without having to speak louder. Students can configure their hearables so that they can better understand the teacher and filter out disruptive classmates.

[0257] For example, the teacher and students are located in defined areas within a closed space (this is the rule). If all devices are coupled to each other, their relative positions are interchangeable, which simplifies source separation. The selected sound source is acoustically emphasized until the user (teacher / student) cancels the selection or until a warning signal interrupts the device's function.

[0258] For example, a teacher or speaker presents content and the device suppresses distracting noise. The teacher wants to hear a student's question and (automatically or via suitable control possibilities) changes the focus of the hearable to the person with the question. After the communication, all sounds are suppressed again. Furthermore, it may be provided that students who feel, for example, that they are being disturbed by their classmates can acoustically adjust them. For example, a student sitting far away from the teacher may have the teacher's voice amplified.

[0259] Alternatively, for example, the teacher and student devices may be combined. The selectivity of the student devices may be temporarily controlled via the teacher device. For particularly important content, the teacher changes the selectivity of the student devices to amplify their voice.

[0260] Another use case of another embodiment concerns the military and soldiers. On the one hand, verbal communication between soldiers in the field occurs via radio, and on the other hand, via voice and direct contact. Radio is mostly used when communication occurs between different units and subgroups. A specific radio etiquette is often used. Voice and direct contact are mostly used to communicate within a unit or group. During a soldier's mission, difficult acoustic conditions (e.g., people screaming, weapon noise, bad weather) can exist that can impair both communication paths. Radio devices with earphones are often part of a soldier's equipment. In addition to providing sound reproduction, they also provide protection against higher levels of sound pressure. These devices are often equipped with microphones to bring environmental signals to the carrier's ear. Active noise suppression is also part of such systems. Range enhancement / extension allows soldiers to voice out and direct contact in noisy environments through intelligent attenuation of interfering noise and selective emphasis of sound through directional reproduction. For this purpose, the soldier's relative position within the room / field must be known. Furthermore, the voice signal and interfering noise must be separated from each other spatially and by content. The system must be able to handle high SNR levels, from low whispers to screams and explosions alike. The advantages of such a system include: verbal communication between soldiers in noisy environments, preservation of hearing protection, the possibility of waiving radio etiquette, and interception security (since it is not a wireless solution).

[0261] For example, voice and direct contact between soldiers on a mission can be complicated by interfering noise. This problem is currently being addressed by short-range and longer-range wireless solutions. New systems enable voice and direct contact in the near field through intelligent spatial emphasis of each speaker and attenuation of ambient noise.

[0262] For example, a soldier is on duty. Voices and sounds are automatically detected and the system amplifies them with simultaneous attenuation of background noise. The system adapts spatial audio playback to ensure full understanding of the target sound source.

[0263] Or, for example, the system can know which soldiers are in a group and only pass through the audio signals of those group members.

[0264] A further use case of a further embodiment relates to security guards and guards. Thus, for example, hearables can be used to disrupt major events (celebrations, demonstrations) for preemptive detection of crimes. The selectivity of the hearable is controlled by keywords, for example calls for help or calls for violence. This presupposes content analysis of the audio signal (e.g., speech recognition).

[0265] For example, a security guard is surrounded by many loud sound sources, and the guard and all sound sources may be moving. A person calling for help may not be able to hear them under normal hearing conditions or may only hear them to a limited extent (poor SNR). A manually or automatically selected sound source is acoustically emphasized until the user cancels the selection. Optionally, a virtual sound object is placed at the location / direction of the sound source of interest to make its location (e.g., for a one-off call for help) easy to find.

[0266] For example, the hearable detects a sound source that may be a potential danger. The guard selects (e.g., by selecting on a tablet) which sound source or event they want to follow. The hearable then adapts the audio playback so that the sound source can be well understood and localized even in the presence of noise interference.

[0267] Alternatively, for example, if the target sound source is silent, a localization signal may be positioned towards / within the distance of the sound source.

[0268] Another use case of another embodiment is on-stage communication, which concerns musicians. On stage, in a rehearsal or concert (e.g., band, orchestra, chorus, music), a single instrument (group) may be inaudible due to difficult acoustic conditions, even if it is still audible in the rest of the environment. This impairs dialogue, as important (accompanying) sounds are no longer perceptible. Hearables can highlight these sounds and make them audible again, thus improving or ensuring dialogue for individual musicians. This use can reduce noise exposure for individual musicians, preventing hearing loss by attenuating, for example, drums, and also allow musicians to hear all important sounds simultaneously.

[0269] For example, a musician without hearables can no longer hear at least one other voice on stage. In this case, hearables may be used. After the rehearsal or concert is over, the user turns off the hearables and then removes them.

[0270] In one example, a user turns on the hearable. The user selects one or more desired instruments to be amplified. When creating music together, the selected musical instruments are amplified and thus made audible again by the hearable. After creating the music, the user turns the hearable off again.

[0271] In another example, a user turns on a hearable. The user selects a desired instrument for which they want to lower the volume. When creating music together, the volume of the selected instrument is reduced by the hearable so that the user can only hear it at a moderate volume.

[0272] For example, an instrument profile can be stored in a hearable.

[0273] Another use case for a further embodiment is source separation as a software module for hearing devices in the ecosystem sense, which concerns manufacturers of hearing devices or users of hearing devices. Manufacturers can use source separation as an additional tool for their hearing devices and offer it to their customers. Thus, hearing devices can also benefit from the development. Licensing models for other markets / devices (headphones, mobile phones, etc.) are also conceivable.

[0274] For example, a user of a hearing device may have difficulty separating different sound sources in a complex hearing situation, for example, to focus on a specific speaker. To enable selective listening without an external additional system (e.g., signal transmission from a mobile radio set via Bluetooth, selective signal transmission in a classroom via FM equipment or induction hearing equipment), the user uses a hearing device with additional functionality for selective listening. Thus, without any external effort, the user can focus on individual sound sources through sound source separation. Finally, the user can turn off the additional functionality and continue listening normally with the hearing device.

[0275] For example, a hearing device user acquires a new hearing device with an integrated additional feature for selective hearing. The user configures the hearing device with the feature for selective hearing. The user then selects a profile (e.g., amplify the loudest / closest sources, amplify speech recognition of specific sounds in their personal surroundings (e.g., UC-CE5-Major Events, etc.)). The hearing device amplifies each sound source according to the configured profile, simultaneously suppressing background noise as requested, and the hearing device user hears individual sound sources from a complex auditory scene, not just the "noise" / clutter of sound sources.

[0276] Alternatively, a user of a hearing device acquires an add-on for selective hearing, such as software for their hearing device. The user installs the add-on on their hearing device. Then, the user configures the selective hearing feature on their hearing device. The user selects a profile (amplifying the loudest / closest sound sources and amplifying speech recognition of specific sounds from the individual's surroundings (e.g., UC-CE5-Major Events)), and the hearing device amplifies each sound source according to the configured profile, while simultaneously suppressing background noise as required. In this case, the user of the hearing device hears individual sound sources from a complex auditory scene, not just the clutter of "noise" / sound sources.

[0277] For example, a hearable can provide a memorizable voice profile.

[0278] A further use case of a further embodiment is professional sports and concerns athletes in competitions. In sports such as biathlon, triathlon, cycling, marathon, etc., professional athletes rely on coaching information or communication with teammates. However, there are situations where they want to protect themselves from loud sounds (shooting in biathlon, loud applause, party horns, etc.) to be able to concentrate. The hearable can be adapted to the respective sport / athlete to allow a fully automatic selection of the relevant sound source (detection of specific voices, volume limiting for typical distracting noises).

[0279] For example, users are very mobile and the type of disturbing noise depends on the sport. Due to intense physical tension, the athlete's control of the device is either impossible or only limited. However, in most sports there are defined procedures (biathlon: running, shooting) and important communication partners (trainers, teammates) can be predefined. Noise is suppressed in general or during specific phases of the activity. Communication between the athlete and their teammates and coaches is always emphasized.

[0280] For example, athletes use hearables that are specially tuned for their type of sport, and the hearables fully automatically (pre-tuned) suppress distracting noise, especially in situations requiring high levels of attention in each type of sport. In addition, the hearables fully automatically (pre-tuned) enhance the trainer and team members when they are within hearing range.

[0281] A further use case of further embodiments is auditory training, for music students, professional musicians, and hobbyist musicians. In music rehearsals (e.g., in orchestras, bands, ensembles, music classes), hearables can be used selectively to track individual voices in a filtered manner. Particularly at the beginning of a rehearsal, it is useful to listen to the final recording of a piece and track your own voice. In some compositions, hearing only foreground sounds does not adequately capture background sounds. Hearables can selectively emphasize sounds based on instrument, etc., for more targeted practice.

[0282] Music students (who wish to do so) can also use hearables to train their auditory abilities and prepare for exams selectively by gradually minimizing individual accentuation until they can eventually extract individual sounds from complex pieces without aid.

[0283] A further possible use case is karaoke, for example when Singstar or similar is not available nearby: vocals can be suppressed from the music if desired, to hear only the instrumental version for signing karaoke.

[0284] For example, a musician may begin learning the sounds from a song by listening to a recording of the music via a playback medium such as a CD player. When the user is done practicing, they turn the hearable off again.

[0285] In one example, a user turns on the hearable. They select the desired instrument they want to amplify. When listening to music, the hearable amplifies the sound of the musical instrument and reduces the volume of the remaining musical instruments, thus allowing the user to better track their own voice.

[0286] In another example, a user turns on the hearable. They select the desired instrument they want to suppress. When listening to a song, the audio of the selected song is suppressed, leaving only the remaining audio to be heard. Thus, the user can practice their instrument with other audio without being distracted by audio from the recording.

[0287] In an embodiment, the hearable may provide stored instrument profiles.

[0288] Another use case for another embodiment is safety at work, relating to workers in noisy environments: Workers in noisy environments such as machine halls or construction sites must protect themselves from noise, but also be able to perceive warning signals and communicate with their colleagues.

[0289] For example, the user may be located in a very loud environment, and the target sound source (warning signal, colleague) may be significantly softer than the interfering noise. The user may be mobile; however, the interfering noise is often stationary. Similar to hearing protection, the noise is permanently reduced and the hearable automatically emphasizes the warning signal. Communication with colleagues is ensured by the amplification of the speaker sound source.

[0290] For example, a user is at work and uses a hearable as hearing protection: a warning signal (e.g., a fire alarm) is acoustically emphasized, causing the user to stop working if necessary.

[0291] Or, for example, a user is at work and uses a hearable as hearing protection, and when there is a need to communicate with a colleague, a communication partner is selected and acoustically emphasized with the help of a suitable interface (here, for example, eye control).

[0292] Another use case of a further embodiment is source separation as a software module for a live translator, and for users of the live translator. The live translator translates spoken foreign languages ​​in real time and can benefit from an upstream software module for source separation. In particular, when multiple speakers are present, the software module can extract the target speaker, potentially improving the translation.

[0293] For example, the software module is part of a live translator (a dedicated device or app on a smartphone). For example, the user can select a target speaker via the device's display. Advantageously, the user and target sound source do not move or move only slightly during translation. The selected sound source position is acoustically emphasized, thus potentially improving the translation.

[0294] For example, a user may wish to converse in a foreign language or listen to a speaker of the foreign language. The user selects the target speaker via an appropriate interface (e.g., a GUI on the display) and the software module optimizes the recording for further use in the translator.

[0295] A further use case of another embodiment is safety in rescue operations, involving firefighters, civil protection, police, and emergency services. For mitigation forces, good communication is essential to successfully complete their missions. Despite high ambient noise, hearing protection is often not possible because relief pressure makes communication impossible. For example, firefighters must be able to accurately transmit and understand commands despite loud motor noise, which is partially generated via radio. Thus, relief forces are exposed to loud noises that prevent them from complying with hearing protection regulations. On the one hand, hearables provide hearing protection for relief forces, while on the other hand, they still allow communication between relief forces. Furthermore, with the help of hearables, relief forces are not acoustically isolated from the environment when carrying their helmets / protective equipment and can therefore provide better support. They can communicate better and also better estimate their own danger (e.g., hear the type of fire occurring).

[0296] For example, a user may be subjected to strong ambient noise and therefore be unable to wear hearing protection, yet still need to be able to communicate with others, using a hearable. After the mission is completed, or the dangerous situation has ended, the user removes the hearable again.

[0297] For example, a user wears a hearable during a mission. The hearable is turned on. The hearable suppresses background noise and amplifies the speech of nearby colleagues and other speakers (e.g., fire victims).

[0298] Alternatively, a user can wear the hearable while on duty, turn it on, and the hearable will suppress background noise and amplify the voices of their colleagues via the radio.

[0299] Where applicable, the hearable is specially designed to meet structural compatibility for operation according to operational specifications. In some cases, the hearable includes an interface to a wireless device.

[0300] While some aspects are described in the context of a device, it will be understood that the aspects also represent a description of a corresponding method, such that a block or structural element of a device should also be understood as a corresponding method step or feature of a method step. Similarly, aspects described in the context of or as a method step also represent a description of a corresponding block or detail or feature of a corresponding device. Some or all of the method steps may be performed using hardware devices such as a microprocessor, a programmable computer, or electronic circuitry. In some embodiments, some or all of the most important method steps may be performed by such devices.

[0301] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation may be performed using a digital storage medium, such as a floppy disk, a DVD, a Blu-ray disk, a CD, a ROM, a PROM, an EPROM, an EEPROM or flash memory, a hard disk, or any other magnetic or optical memory on which electronically readable control signals are stored that cooperate or can cooperate with a programmable computer system to perform the respective method. This is why the digital storage medium may be computer-readable.

[0302] Thus, some embodiments according to the present invention include a data carrier comprising electronically readable control signals capable of cooperating with a programmable computer system to perform any of the methods described herein.

[0303] In general, embodiments of the present invention may be implemented as a computer program product having program code that is effective to perform any of the methods when the computer program product is run on a computer.

[0304] The program code may for example be stored on a machine readable carrier.

[0305] Other embodiments comprise the computer program for performing any of the methods described herein, said computer program stored on a machine readable carrier. In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

[0306] A further embodiment of the inventive methods is, therefore, a data carrier (or digital storage medium or computer readable medium) having recorded thereon a computer program for performing any of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible or non-volatile.

[0307] A further embodiment of the inventive method is, therefore, a data stream or sequence of signals representing the computer program for performing any of the methods described herein, The data stream or sequence of signals can for example be adapted to be transmitted via a data communications link, for example via the Internet.

[0308] Further embodiments comprise a processing unit, for example a computer or programmable logic device, configured to or adapted to perform any of the methods described herein.

[0309] A further embodiment comprises a computer having installed thereon the computer program for performing any of the methods described herein.

[0310] Further embodiments according to the present invention include a device or system configured to transmit a computer program for performing at least one of the methods described herein to a receiver. The transmission may be, for example, electronic or optical. The receiver may be, for example, a computer, a mobile device, a memory device, or a similar device. The device or system may, for example, include a file server for transmitting the computer program to the receiver.

[0311] In some embodiments, a programmable logic device (e.g., a field programmable gate array, FPGA) can be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array can cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are performed in some embodiments by any hardware device. The hardware device may be any universally applicable hardware, such as a computer processor (CPU), or method-specific hardware, such as an ASIC.

[0312] The above-described embodiments merely represent illustrative of the principles of the present invention. It is understood that others skilled in the art will appreciate modifications and variations of the arrangements and details described herein. Therefore, it is intended that the present invention be limited only by the scope of the following claims and not by the specific details presented herein by way of description and discussion of the embodiments.

[0313] References [1] V.Valimaki,A.Franck,J.Ramo,H.Gamper,and L.Savioja,”Assisted listening using a headset:Enhancing audio perception in real,augmented,and virtual environments,” IEEE Signal Processing Magazine,volume 32,no.2,pp.92-99,March 2015

[0314] [2] K.Brandenburg,E.Cano,F.Klein,T.Kollmer,H.Lukashevich,A.Neidhardt,U.Sloma,and S.Werner,”Plausible augmentation of auditory scenes using dynamic binaural synthesis for personalized auditory realities,” in Proc.of AES International Conference on Audio for Virtual and Augmented Reality,August 2018

[0315] [3] S.Argentieri,P.Dans,and P.Soures,”A survey on sound source localization in robotics:From binaural to array processing methods,” Computer Speech Language,volume 34,no.1,pp.87-112,2015

[0316] [4] D.FitzGerald,A.Liutkus,and R.Badeau,”Projection-based demixing of spatial audio,” IEEE / ACM Trans.on Audio,Speech,and Language Processing,volume 24,no.9,pp.1560-1572,2016

[0317] [5] E.Cano,D.FitzGerald,A.Liutkus,M.D.Plumbley,and F.Stoter,”Musical source separation:An introduction,” IEEE Signal Processing Magazine,volume 36,no.1,pp.31-40,January 2019

[0318] [6] S.Gannot,E.Vincent,S.Markovich-Golan,and A.Ozerov,”A consolidated perspective on multimicrophone speech enhancement and source separation,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 25,no.4,pp.692-730,April 2017

[0319] [7] E.Cano,J.Nowak,and S.Grollmisch,”Exploring sound source separation for acoustic condition monitoring in industrial scenarios,” in Proc.of 25th European Signal Processing Conference(EUSIPCO),August 2017,pp.2264-2268

[0320] [8] T.Gerkmann,M.Krawczyk-Becker,and J.Le Roux,”Phase processing for single-channel speech enhancement:History and recent advances,” IEEE Signal Processing Magazine,volume 32,no.2,pp.55-66,March 2015

[0321] [9] E.Vincent,T.Virtanen,and S.Gannot,Audio Source Separation and Speech Enhancement.Wiley,2018

[0322]

[10] D.Matz,E.Cano,and J.Abesser,”New sonorities for early jazz recordings using sound source separation and automatic mixing tools,” in Proc.of the 16th International Society for Music Information Retrieval Conference.Malaga,Spain:ISMIR,October 2015,pp.749-755

[0323]

[11] S.M.Kuo and D.R.Morgan,”Active noise control:a tutorial review,” Proceedings of the IEEE,volume 87,no.6,pp.943-973,June 1999

[0324]

[12] A.McPherson,R.Jack,and G.Moro,”Action-sound latency:Are our tools fast enough?” in Proceedings of the International Conference on New Interfaces for Musical Expression,July 2016

[0325]

[13] C.Rottondi,C.Chafe,C.Allocchio,and A.Sarti,”An overview on networked music performance technologies,” IEEE Access,volume 4,pp.8823-8843,2016

[0326]

[14] S.Liebich,J.Fabry,P.Jax,and P.Vary,”Signal processing challenges for active noise cancellation headphones,” in Speech Communication;13th ITG-Symposium,October 2018,pp.1-5

[0327]

[15] E.Cano,J.Liebetrau,D.Fitzgerald,and K.Brandenburg,”The dimensions of perceptual quality of sound source separation,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2018,pp.601-605

[0328]

[16] P.M.Delgado and J.Herre,”Objective assessment of spatial audio quality using directional loudness maps,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),May 2019,pp.621-625

[0329]

[17] C.H.Taal,R.C.Hendriks,R.Heusdens,and J.Jensen,”An algorithm for intelligibility prediction of time-frequency weighted noisy speech,” IEEE Transactions on Audio,Speech,and Language Processing,volume 19,no.7,pp.2125-2136,September 2011

[0330]

[18] M.D.Plumbley,C.Kroos,J.P.Bello,G.Richard,D.P.Ellis,and A.Mesaros,Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop(DCASE2018).Tampere University of Technology.Laboratory of Signal Processing,2018

[0331]

[19] R.Serizel,N.Turpault,H.Eghbal-Zadeh,and A.Parag Shah,”Large-Scale Weakly Labeled Semi-Supervised Sound Event Detection in Domestic Environments,” July 2018,submitted to DCASE2018 Workshop

[0332]

[20] L.JiaKai,”Mean teacher convolution system for dcase 2018 task 4,” DCASE2018 Challenge,Tech.Rep.,September 2018

[0333]

[21] G.Parascandolo,H.Huttunen,and T.Virtanen,”Recurrent neural networks for polyphonic sound event detection in real life recordings,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),March 2016,pp.6440-6444

[0334]

[22] E.C,Cakir and T.Virtanen,”End-to-end polyphonic sound event detection using convolutional recurrent neural networks with learned time-frequency representation input,” in Proc.of International Joint Conference on Neural Networks(IJCNN),July 2018,pp.1-7

[0335]

[23] Y.Xu,Q.Kong,W.Wang,and M.D.Plumbley,”Large-Scale Weakly Supervised Audio Classification Using Gated Convolutional Neural Network,” in Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),Calgary,AB,Canada,2018,pp.121-125

[0336]

[24] B.Frenay and M.Verleysen,”Classification in the presence of label noise:A survey,” IEEE Transactions on Neural Networks and Learning Systems,volume 25,no.5,pp.845-869,May 2014

[0337]

[25] E.Fonseca,M.Plakal,D.P.W.Ellis,F.Font,X.Favory,and X.Serra,”Learning sound event classifiers from web audio with noisy labels,” in Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),Brighton,UK,2019

[0338]

[26] M.Dorfer and G.Widmer,”Training general-purpose audio tagging networks with noisy labels and iterative self-verification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop(DCASE2018),Surrey,UK,2018

[0339]

[27] S.Adavanne,A.Politis,J.Nikunen,and T.Virtanen,”Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,” IEEE Journal of Selected Topics in Signal Processing,pp.1-1,2018

[0340]

[28] Y.Jung,Y.Kim,Y.Choi,and H.Kim,”Joint learning using denoising variational autoencoders for voice activity detection,” in Proc.of Interspeech,September 2018,pp.1210-1214

[0341]

[29] F.Eyben,F.Weninger,S.Squartini,and B.Schuller,”Real-life voice activity detection with LSTM recurrent neural networks and an application to hollywood movies,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing,May 2013,pp.483-487

[0342]

[30] R.Zazo-Candil,T.N.Sainath,G.Simko,and C.Parada,”Feature learning with raw-waveform CLDNNs for voice activity detection,” in Proc.of INTERSPEECH,2016

[0343]

[31] M.McLaren,Y.Lei,and L.Ferrer,”Advances in deep neural network approaches to speaker recognition,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2015,pp.4814-4818

[0344]

[32] D.Snyder,D.Garcia-Romero,G.Sell,D.Povey,and S.Khudanpur,”X-vectors:Robust DNN embeddings for speaker recognition,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2018,pp.5329-5333

[0345]

[33] M.McLaren,D.Castan,M.K.Nandwana,L.Ferrer,and E.Yilmaz,”How to train your speaker embeddings extractor,” in Odyssey,2018

[0346]

[34] S.O.Sadjadi,J.W.Pelecanos,and S.Ganapathy,”The IBM speaker recognition system:Recent advances and error analysis,” in Proc.of Interspeech,2016,pp.3633-3637

[0347]

[35] Y.Han,J.Kim,and K.Lee,”Deep convolutional neural networks for predominant instrument recognition in polyphonic music,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 25,no.1,pp.208-221,January 2017

[0348]

[36] V.Lonstanlen and C.-E.Cella,”Deep convolutional networks on the pitch spiral for musical instrument recognition,” in Proceedings of the 17th International Society for Music Information Retrieval Conference.New York,USA:ISMIR,2016,pp.612-618

[0349]

[37] S.Gururani,C.Summers,and A.Lerch,”Instrument activity detection in polyphonic music using deep neural networks,” in Proceedings of the 19th International Society for Music Information Retrieval Conference.Paris,France:ISMIR,September 2018,pp.569-576

[0350]

[38] J.Schlutter and B.Lehner,”Zero mean convolutions for level-invariant singing voice detection,” in Proceedings of the 19th International Society for Music Information Retrieval Conference.Paris,France:ISMIR,September 2018,pp.321-326

[0351]

[39] S.Delikaris-Manias,D.Pavlidi,A.Mouchtaris,and V.Pulkki,”DOA estimation with histogram analysis of spatially constrained active intensity vectors,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),March 2017,pp.526-530

[0352]

[40] S.Chakrabarty and E.A.P.Habets,”Multi-speaker DOA estimation using deep convolutional networks trained with noise signals,” IEEE Journal of Selected Topics in Signal Processing,volume 13,no.1,pp.8-21,March 2019

[0353]

[41] X.Li,L.Girin,R.Horaud,and S.Gannot,”Multiple-speaker localization based on direct-path features and likelihood maximization with spatial sparsity regularization,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 25,no.10,pp.1997-2012,October 2017

[0354]

[42] F.Grondin and F.Michaud,”Lightweight and optimized sound source localization and tracking methods for open and closed microphone array configurations,” Robotics and Autonomous Systems,volume 113,pp.63-80,2019

[0355]

[43] D.Yook,T.Lee,and Y.Cho,”Fast sound source localization using two-level search space clustering,” IEEE Transactions on Cybernetics,volume 46,no.1,pp.20-26,January 2016

[0356]

[44] D.Pavlidi,A.Griffin,M.Puigt,and A.Mouchtaris,”Real-time multiple sound source localization and counting using a circular microphone array,” IEEE Transactions on Audio,Speech,and Language Processing,volume 21,no.10,pp.2193-2206,October 2013

[0357]

[45] P.Vecchiotti,N.Ma,S.Squartini,and G.J.Brown,”End-to-end binaural sound localisation from the raw waveform,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),May 2019,pp.451-455

[0358]

[46] Y.Luo,Z.Chen,and N.Mesgarani,”Speaker-independent speech separation with deep attractor network,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 26,no.4,pp.787-796,April 2018

[0359]

[47] Z.Wang,J.Le Roux,and J.R.Hershey,”Multi-channel deep clustering:Discriminative spectral and spatial embeddings for speaker-independent speech separation,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2018,pp.1-5

[0360]

[48] G.Naithani,T.Barker,G.Parascandolo,L.BramslLw,N.H.Pontoppidan,and T.Virtanen,”Low latency sound source separation using convolutional recurrent neural networks,” in Proc.of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics(WASPAA),October 2017,pp.71-75

[0361]

[49] M.Sunohara,C.Haruta,and N.Ono,”Low-latency real-time blind source separation for hearing aids based on time-domain implementation of online independent vector analysis with truncation of non-causal components,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),March 2017,pp.216-220

[0362]

[50] Y.Luo and N.Mesgarani,”TaSNet:Time-domain audio separation network for real-time,single-channel speech separation,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),April 2018,pp.696-700

[0363]

[51] J.Chua,G.Wang,and W.B.Kleijn,”Convolutive blind source separation with low latency,” in Proc.of IEEE International Workshop on Acoustic Signal Enhancement(IWAENC),September 2016,pp.1-5

[0364]

[52] Z.Rafii,A.Liutkus,F.Stoter,S.I.Mimilakis,D.FitzGerald,and B.Pardo,”An overview of lead and accompaniment separation in music,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 26,no.8,pp.1307-1335,August 2018

[0365]

[53] F.-R.Stoter,A.Liutkus,and N.Ito,”The 2018 signal separation evaluation campaign,” in Latent Variable Analysis and Signal Separation,Y.Deville,S.Gannot,R.Mason,M.D.Plumbley,and D.Ward,Eds.Cham:Springer International Publishing,2018,pp.293-305

[0366]

[54] J.-L.Durrieu,B.David,and G.Richard,”A musically motivated midlevel representation for pitch estimation and musical audio source separation,” Selected Topics in Signal Processing,IEEE Journal of,volume 5,no.6,pp.1180-1191,October 2011

[0367]

[55] S.Uhlich,M.Porcu,F.Giron,M.Enenkl,T.Kemp,N.Takahashi,and Y.Mitsufuji,”Improving music source separation based on deep neural networks through data augmentation and network blending,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),2017

[0368]

[56] P.N.Samarasinghe,W.Zhang,and T.D.Abhayapala,”Recent advances in active noise control inside automobile cabins:Toward quieter cars,” IEEE Signal Processing Magazine,volume 33,no.6,pp.61-73,November 2016

[0369]

[57] S.Papini,R.L.Pinto,E.B.Medeiros,and F.B.Coelho,”Hybrid approach to noise control of industrial exhaust systems,” Applied Acoustics,volume 125,pp.102-112,2017

[0370]

[58] J.Zhang,T.D.Abhayapala,W.Zhang,P.N.Samarasinghe,and S.Jiang,”Active noise control over space:A wave domain approach,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 26,no.4,pp.774-786,April 2018

[0371]

[59] X.Lu,Y.Tsao,S.Matsuda,and C.Hori,”Speech enhancement based on deep denoising autoencoder,” in Proc.of Interspeech,2013

[0372]

[60] Y.Xu,J.Du,L.Dai,and C.Lee,”A regression approach to speech enhancement based on deep neural networks,” IEEE / ACM Transactions on Audio,Speech,and Language Processing,volume 23,no.1,pp.7-19,January 2015

[0373]

[61] S.Pascual,A.Bonafonte,and J.Serra,”SEGAN:speech enhancement generative adversarial network,” in Proc.of Interspeech,August 2017,pp.3642-3646

[0374]

[62] F.Weninger,H.Erdogan,S.Watanabe,E.Vincent,J.Le Roux,J.R.Hershey,and B.Schuller,”Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,” in Latent Variable Analysis and Signal Separation,E.Vincent,A.Yeredor,Z.Koldovsky,and P.Tichavsky,Eds.Cham:Springer International Publishing,2015,pp.91-99

[0375]

[63] H.Wierstorf,D.Ward,R.Mason,E.M.Grais,C.Hummersone,and M.D.Plumbley,”Perceptual evaluation of source separation for remixing music,” in Proc.of Audio Engineering Society Convention 143,October 2017

[0376]

[64] J.Pons,J.Janer,T.Rode,and W.Nogueira,”Remixing music using source separation algorithms to improve the musical experience of cochlear implant users,” The Journal of the Acoustical Society of America,volume 140,no.6,pp.4338-4349,2016

[0377]

[65] Q.Kong,Y.Xu,W.Wang,and M.D.Plumbley,”A joint separation-classification model for sound event detection of weakly labelled data,” in Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),March 2018

[0378]

[66] T.v.Neumann,K.Kinoshita,M.Delcroix,S.Araki,T.Nakatani,and R.Haeb-Umbach,”All-neural online source separation,counting,and diarization for meeting analysis,” in Proc.of IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),May 2019,pp.91-95

[0379]

[67] S.Gharib,K.Drossos,E.Cakir,D.Serdyuk,and T.Virtanen,”Unsupervised adversarial domain adaptation for acoustic scene classification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop(DCASE),November 2018,pp.138-142

[0380]

[68] A.Mesaros,T.Heittola,and T.Virtanen,”A multi-device dataset for urban acoustic scene classification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop,Surrey,UK,2018

[0381]

[69] J.Abesser,M.Gotze,S.Kuhnlenz,R.Grafe,C.Kuhn,T.Clauss,H.Lukashevich,”A Distributed Sensor Network for Monitoring Noise Level and Noise Sources in Urban Environments,” in Proceedings of the 6th IEEE International Conference on Future Internet of Things and Cloud(FiCloud),Barcelona,Spain,pp.318-324.,2018

[0382]

[70] T.Virtanen,M.D.Plumbley,D.Ellis(Eds.),”Computational Analysis of Sound Scenes and Events,” Springer,2018

[0383]

[71] J.Abesser,S.Ioannis Mimilakis,R.Grafe,H.Lukashevich,”Acoustic scene classification by combining autoencoder-based dimensionality reduction and convolutional neural net-works,” in Proceedings of the 2nd DCASE Workshop on Detection and Classification of Acoustic Scenes and Events,Munich,Germany,2017

[0384]

[72] A.Avni,J.Ahrens,M.Geierc,S.Spors,H.Wierstorf,B.Rafaely,”Spatial perception of sound fields recorded by spherical microphone arrays with varying spatial resolution,” Journal of the Acoustic Society of America,133(5),pp.2711-2721,2013

[0385]

[73] E.Cano,D.FitzGerald,K.Brandenburg,”Evaluation of quality of sound source separation algorithms:Human perception vs quantitative metrics,” in Proceedings of the 24th European Signal Processing Conference(EUSIPCO),pp.1758-1762,2016

[0386]

[74] S.Marchand,”Audio scene transformation using informed source separation,” The Journal of the Acoustical Society of America,140(4),p.3091,2016

[0387]

[75] S.Grollmisch,J.Abesser,J.Liebetrau,H.Lukashevich,”Sounding industry:Challenges and datasets for industrial sound analysis(ISA),” in Proceedings of the 27th European Signal Processing Conference(EUSIPCO)(submitted),A Coruna,Spain,2019

[0388]

[76] J.Abesser,M.Muller,”Fundamental frequency contour classification:A comparison between hand-crafted and CNN-based features,” in Proceedings of the 44th IEEE International Conference on Acoustics,Speech,and Signal Processing(ICASSP),2019

[0389]

[77] J.Abesser,S.Balke,M.Muller,”Improving bass saliency estimation using label propagation and transfer learning,” in Proceedings of the 19th International Society for Music Information Retrieval Conference(ISMIR),Paris,France,pp.306-312,2018

[0390]

[78] C.-R.Nagar,J.Abesser,S.Grollmisch,”Towards CNN-based acoustic modeling of seventh chords for recognition chord recognition,” in Proceedings of the 16th Sound&Music Computing Conference(SMC)(submitted),Malaga,Spain,2019

[0391]

[79] J.S.Gomez,J.Abesser,E.Cano,”Jazz solo instrument classification with convolutional neural networks,source separation,and transfer learning”,in Proceedings of the 19th International Society for Music Information Retrieval Conference(ISMIR),Paris,France,pp.577-584,2018

[0392]

[80] J.R.Hershey,Z.Chen,J.Le Roux,S.Watanabe,”Deep clustering:Discriminative embeddings for segmentation and separation,” in Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),pp.31-35,2016

[0393]

[81] E.Cano,G.Schuller,C.Dittmar,”Pitch-informed solo and accompaniment separation towards its use in music education applications”,EURASIP Journal on Advances in Signal Processing,2014:23,pp.1-19

[0394]

[82] S.I.Mimilakis,K.Drossos,J.F.Santos,G.Schuller,T.Virtanen,Y.Bengio,”Monaural Singing Voice Separation with Skip-Filtering Connections and Recurrent Inference of Time-Frequency Mask,” in Proceedings of the IEEE International Conference on Acoustics,Speech,and Signal Processing(ICASSP),Calgary,Canada,S.721-725,2018

[0395]

[83] J.F.Gemmeke,D.P.W.Ellis,D.Freedman,A.Jansen,W.Lawrence,R.C.Moore,M.Plakal,M.Ritter,”Audio Set:An ontology and human-labeled dataset for audio events,” in Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),New Orleans,USA,2017

[0396]

[84] Kleiner,M.”Acoustics and Audio Technology,”.3rd ed.USA:J.Ross Publishing,2012

[0397]

[85] M.Dickreiter,V.Dittel,W.Hoeg,M.Wohr,M.,,Handbuch der Tonstudiotechnik,” A.medienakademie(Eds).7th edition,Vol.1.,Munich:K.G.Saur Verlag,2008

[0398]

[86] F.Muller,M.Karau.,,Transparant hearing,” in:CHI ,02 Extended Abstracts on Human Factors in Computing Systems(CHI EA ’02),Minneapolis,USA,pp.730-731,April 2002

[0399]

[87] L.Vieira.”Super hearing:a study on virtual prototyping for hearables and hearing aids,” Master Thesis,Aalborg University,2018.Available:https: / / projekter.aau.dk / projekter / files / 287515943 / MasterThesis_Luis.pdf

[0400]

[88] Sennheiser,”AMBEO Smart Headset,” [Online].Available: https: / / de-de.sennheiser.com / finalstop [Accessed:March 1,2019]

[0401]

[89] Orosound ”Tilde Earphones” [Online].Available: https: / / www.orosound.com / tilde-earphones / [Accessed;March 1,2019]

[0402]

[90] Brandenburg, K., Cano Ceron, E., Klein, F., Kollmer, T., Lukashevich, H., Neidhardt, A., Nowak, J., Sloma, U., and Werner, S., "Personalized auditory reality," in 44th Annual Meeting for Acoustics (DAGA), Garching near Munich, German Acoustical Society (DEGA), 2018

[0403]

[91] US 2015 195641 A1,Application date:January 6,2014;published on July 9,2015.

Claims

1. an analyzer (152) for determining a plurality of binaural room impulse responses; a loudspeaker signal generator (154) for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source; Equipped with the at least two loudspeaker signals are played by a user wearing headphones; the headphones include two headphone capsules, each of which is provided with at least one microphone for measuring sound in the two headphone capsules; the analyzer (152) is configured to determine the plurality of binaural room impulse responses using the measurements of the at least one microphone in each of the two headphone capsules; the at least one microphone in each of the two headphone capsules is configured to generate one or more recordings of a sound situation in a reproduction room prior to playback of the at least two loudspeaker signals by the headphones, determine an estimate of an audio signal from at least one sound source from the one or more recordings, and determine a binaural room impulse response of the plurality of binaural room impulse responses for the sound source in the reproduction room; the user listens to the listening environment through the headphones, and the headphones play an augmented signal to play the additional sound source; the at least one microphone in each of the two headphone capsules is configured to: generate, during playback of the at least two loudspeaker signals by the headphones, one or more further recordings of the sound situation in the reproduction room; subtract the augmented signal from these one or more further recordings; determine the estimates of the sound signals from one or more sound sources; and determine the binaural room impulse responses of the plurality of binaural room impulse responses for the sound sources in the reproduction room. system.

2. The system comprises the headphones, the headphones are configured to output the at least two loudspeaker signals; The system of claim 1 .

3. the analyzer (152) is configured to determine an acoustic room characteristic of the reproduction room and to adapt the plurality of binaural room impulse responses according to the acoustic room characteristic.

3. The system according to claim 1 or 2.

4. the at least one microphone is disposed in each of the two headphone capsules to measure sounds near the entrance of the ear canal; A system according to any one of claims 1 to 3.

5. the system includes one or more further microphones outside the two headphone capsules for measuring the sound situation in the reproduction room; A system according to any one of claims 1 to 4.

6. the headphones include a headphone band, and at least one of the one or more additional microphones is disposed on the headphone band. The system of claim 5.

7. the loudspeaker signal generator (154) is configured to generate the at least two loudspeaker signals by convolving each of the plurality of binaural room impulse responses with a source signal of a plurality of one or more source signals. A system according to any one of claims 1 to 6.

8. the analyzer (152) is configured to determine at least one of the plurality of binaural room impulse responses in response to a movement of the headphones. A system according to any one of claims 1 to 7.

9. the system includes a sensor for determining movement of the headphones; The system of claim 8.

10. 1. A system for assisting selective hearing, comprising: a detector (110) for detecting source signal portions of one or more sound sources using at least two received microphone signals of the auditory environment; a position determiner (120) that assigns position information to each of the one or more sound sources; Furthermore, said analyzer (152) and said loudspeaker signal generator (154) together form a signal generator (150); the analyzer (152) of the signal generator (150) is configured to generate the plurality of binaural room impulse responses, the plurality of binaural room impulse responses being a plurality of binaural room impulse responses for each of one or more sound sources, the plurality of binaural room impulse responses being dependent on the position information of the sound sources and a user's head orientation; the loudspeaker signal generator (154) of the signal generator (150) is configured to generate the at least two loudspeaker signals in response to the plurality of binaural room impulse responses. The system of claim 1 .

11. the detector (110) is configured to detect the source signal portions of the one or more sound sources by using a deep learning model. The system of claim 10.

12. the location determiner (120) is configured to determine the location information for each of the one or more sound sources in response to captured images or recorded video.

12. The system according to claim 10 or 11.

13. the system further comprises a signal portion modifier (140) configured to select the at least one sound source in response to a previously learned user scenario and to modify the sound source in response to the previously learned user scenario.

13. A system according to any one of claims 10 to 12.

14. The system includes a remote device (190) including the detector (110), the position determiner (120), and the signal generator (150); the remote device is spatially separated from the headphones; 14. A system according to any one of claims 10 to 13.

15. The system of claim 14, wherein the remote device (190) is a smartphone.

16. 1. A method comprising: determining, by an analyzer (152) of the system, a plurality of binaural room impulse responses; generating at least two loudspeaker signals by a loudspeaker signal generator (154) of the system in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source; - playing the at least two loudspeaker signals by a user wearing headphones, the headphones including two headphone capsules, each of the two headphone capsules having at least one microphone disposed therein for measuring sound at each of the two headphone capsules; Including, the analyzer uses the measurements of the at least one microphone in each of the two headphone capsules to determine the plurality of binaural room impulse responses; the at least one microphone in each of the two headphone capsules generates one or more recordings of a sound situation in a reproduction room prior to playback of the at least two loudspeaker signals by the headphones, determines an estimate of an audio signal from at least one sound source from the one or more recordings, and determines a binaural room impulse response of the plurality of binaural room impulse responses for the sound source in the reproduction room; the user listens to the listening environment through the headphones, and the headphones play an augmented signal to play the additional sound source; the at least one microphone in each of the two headphone capsules generates one or more further recordings of the sound situation in the reproduction room during playback of the at least two loudspeaker signals by the headphones, subtracts the augmented signal from these one or more further recordings, determines the estimates of the sound signals from one or more sound sources, and determines the binaural room impulse responses of the plurality of binaural room impulse responses for the sound sources in the reproduction room. method.

17. 17. Computer program having a program code for causing the computer to perform the method according to claim 16 when the computer program is run by a computer or signal processor.

Images