A method for converting audio input data into audio output data and a listening device thereof.

Abstract

A method for converting audio input data into audio output data and a listening device thereof are provided. The method (200) includes receiving (202) audio input data (110), providing (204) background sound data (122) by separating speech components from the audio input data (110) using a speech removal module (120), determining (206) acoustic scene data (126) linked to an acoustic scene (AS) matching the background sound data (122) by using an acoustic scene classifier (ASC) module (124), selecting (208) a dedicated noise reduction (S-NR) module (128A-C) based on the acoustic scene data (126), and processing (210) the audio input data (110) using the dedicated noise reduction (S-NR) module (128A-C) to generate the audio output data (130).

Description

Technical Field

[0001] This invention relates to a listening apparatus and method for processing audio data. More specifically, this disclosure relates to a method for improving speech clarity and a listening apparatus thereof. Background Technology

[0002] In many modern loudspeakers, headphones, and other sound reproduction devices, audio data processing algorithms are used to reduce noise and other unwanted sound signals. For conference loudspeakers and other devices that reproduce speech, different measures can be taken to reduce the impact of sound signals identified as non-speech. For example, sound signal components with characteristic frequencies outside the speech range can be identified as noise. Other factors to consider in noise identification are frequency patterns or repetitions. Once identified as noise, these components can be removed or at least reduced from the audio data before the conference loudspeaker converts the audio data into a sound signal. By using this method, audio data originating from a person speaking into a microphone while sitting on a train can be processed so that the speech components can be distinguished from the train sound components. As a next step, the train sound components can be removed, which has the positive effect that the person listening to the conference loudspeaker will be less or not at all disturbed by the noisy train environment.

[0003] Today, a common method for improving speech intelligibility is to use what's known as a speech detector. In short, by knowing when audio data includes and excludes speech components, different types of audio data processing can be used. For example, when speech is detected, the sound signal within the frequency range associated with the speech can be amplified to provide emphasis on the speech.

[0004] A recent approach to improving speech intelligibility is using what's known as acoustic scene classification. Simply put, instead of detecting the presence of speech as is the general principle when using a speech detector, it analyzes audio data and links it to one of several acoustic scenes. For example, continuing the example above, by analyzing audio data produced by someone speaking while sitting on a train, an acoustic scene classification system can conclude that the acoustic scene linked to that audio data is "train" or something similar. Knowing the acoustic scene, this input can be used to provide algorithms for improving speech intelligibility, resulting in more precise audio data processing.

[0005] Although audio data processing has improved significantly over the past few decades, there is still a need to provide methods and devices that can clearly present speech from noisy environments to users at the receiving end. Summary of the Invention

[0006] According to a first aspect, a computer-implemented method for converting audio input data into audio output data is provided, the method comprising:

[0007] Receive audio input data,

[0008] Background sound data is provided by separating speech components from the audio input data using a speech removal module.

[0009] By using the Acoustic Scene Classifier (ASC) module, acoustic scene data linked to the matching background sound data (AS) is determined.

[0010] Based on acoustic scene data, a dedicated noise reduction (S-NR) module is selected, and

[0011] Audio input data is processed using a dedicated noise reduction (S-NR) module to generate audio output data.

[0012] The advantage of this method is that, by using background sound data independently, the acoustic scene linked to the background sound data can be accurately determined. Once the acoustic scene is determined, an S-NR module specifically configured for that scene can be selected. As a result, speech intelligibility can be improved. Another advantage of this method is that, by separating the speech components, these components will not negatively affect the determination of the acoustic scene data.

[0013] The S-NR module can be a neural network, and the dedicated noise reduction (S-NR) module can be selected from a fixed set of pre-trained neural networks, each pre-trained neural network being designed for a sound environment with specific characteristics.

[0014] By using a fixed set of pre-trained S-NR modules, these modules can be trained with data from different locations. More reliable S-NR modules can be achieved by accessing large amounts of data.

[0015] The S-NR module can be a neural network, such as a convolutional neural network, but other methods can also be used for the S-NR module. For example, the S-NR module can be a statistical model. The S-NR module can be a pre-trained machine learning model.

[0016] A fixed set of dedicated noise reduction (S-NR) modules may include at least three modules, including one module for a transportation environment, one module for an outdoor environment, and one module for an indoor environment.

[0017] It has been found that different acoustic scenarios can be divided into three groups: transportation environments, including, for example, bus, train, and tram transportation; outdoor environments, including, for example, parks, streets, and markets; and indoor environments, including, for example, cafes, shopping malls, restaurants, and airports. With these three modules, transitions from one environment to another can be reliably identified.

[0018] The method can be performed at the receiver (RX) device to receive audio input data, and can also transmit audio input data from the transmitter (TX) device, and may further include:

[0019] Receive time-frequency (TF) masking from the TX device at the RX device.

[0020] A TF mask is provided to the speech removal device arranged in the RX device, so that background sound data is generated by combining the audio input data with the TF mask.

[0021] It has been found that TF masking is advantageous for removing speech components from audio input data, thereby obtaining background sound data. Therefore, instead of using TF masking to remove background sound data from audio input data while preserving speech components (which is a common practice with TF masking), it has been found that TF masking can be used in the reverse manner to remove speech components instead.

[0022] The step of providing background sound data can be performed by multiplying the audio input data with a time-frequency (TF) mask.

[0023] The advantage of multiplication (e.g., element-wise multiplication) is that, from the perspective of computational power, background sound data can be obtained in an efficient manner.

[0024] Audio input data and time-frequency (TF) masking can be received in parallel via the RX device.

[0025] By determining the TF mask in the TX device, the TF mask can be transmitted in parallel with the audio input data to the RX device. Once received in the RX device, the TF mask can be combined with the audio input data, for example, by multiplication. Using this method, less computational power is required in the RX device, which can be advantageous if audio input data is received from multiple TX devices.

[0026] Each of the S-NR modules can be more complex than the receiver-side universal noise reduction (RX G-NR) module, which is configured to identify TF masking based on audio input data, such that the computational power associated with each of the S-NR modules is greater than that associated with the RX G-NR module.

[0027] Better overall performance can be achieved by making the RX G-NR module less computationally complex than the S-NR module. Since the purpose of the RX G-NR module is to identify the most suitable S-NR module (via background sound data and the acoustic scene), it has been found beneficial to allocate more computational power to the S-NR module than to the RX G-NR module, especially when there are only a few (e.g., fewer than ten) different S-NR modules. It is also possible to make the TX G-NR module less computationally complex than the S-NR module.

[0028] The RX G-NR module and TF masking detector can be arranged in the RX device, and the method may also include:

[0029] Determine if TF masking is received from the TX device.

[0030] If TF masking is received via the RX device, disable the RX G-NR module, or

[0031] If TF masking is not received through the RX device, enable the RX G-NR module to provide TF masking through the RX G-NR module.

[0032] By arranging the RX device in this way, it can communicate with a TX device configured to transmit audio input data and TF masking or other types of auxiliary information, or with a TX device that only transmits audio input data. In the latter case, the RX device provides TF masking itself. This flexibility enhances the versatility of the RX device.

[0033] The method may also include:

[0034] TF masking was identified by using the RX G-NR module on the audio input data, and

[0035] Provide TF masking to the speech removal module so that background sound data can be generated by combining audio input data with TF masking.

[0036] The method may also include:

[0037] Receive location data,

[0038] The step of selecting the S-NR module is based on the combination of acoustic scene data and location data.

[0039] By including location data, more precise selection of the acoustic scene can be achieved. For example, if the location data indicates that the TX device is moving above a speed threshold and the location data coincides with the known position of a train track, the location data indicates that the acoustic scene could be a transportation environment. Conversely, if the location data indicates that the TX device is not moving, the likelihood of a transportation environment is lower.

[0040] According to the second aspect, a listening device, such as a conference speaker, is provided, comprising:

[0041] A receiver (RX) device is configured to receive audio input data from a transmitter (TX) device.

[0042] The RX device further includes:

[0043] The speech removal module is configured to provide background sound data by separating speech components from the audio input data.

[0044] The Acoustic Scene Classifier (ASC) module is configured to determine acoustic scene data that is linked to the acoustic scene (AS) that matches the background sound data.

[0045] A dedicated noise reduction (S-NR) selector is configured to select a dedicated noise reduction (S-NR) module based on acoustic scene data, and

[0046] The processing module is configured to process audio input data using a dedicated noise reduction (S-NR) module to generate audio output data.

[0047] The same advantages and features described above regarding the first aspect also apply to this second aspect.

[0048] The RX device may also include:

[0049] The TF masking receiver is configured to receive TF masking from the TX device.

[0050] The speech removal module can be configured to remove speech components from the audio input data by combining the audio input data with TF masking.

[0051] The RX device may also include:

[0052] The receiver universal noise reduction (RX G-NR) module is configured to provide TF masking based on audio input data, and

[0053] The TF masking detector is configured to identify whether TF masking is provided from the TX device, and to disable the RX G-NR module when TF masking is provided through the TX device, or to enable the RX G-NR module when TF masking is not provided through the TX device, so that TF masking is provided through the RX G-NR module.

[0054] The RX device may also include:

[0055] The receiver universal noise reduction (RX G-NR) module is configured to provide TF masking based on audio input data and to provide TF masking to the speech removal module, so that background sound data is generated by combining the audio input data with the TF masking.

[0056] The RX device may also include:

[0057] The location data receiver is configured to receive location data from the TX device.

[0058] The S-NR selector is configured to select an S-NR module based on a combination of acoustic scene data and location data.

[0059] As used herein, the term "listening device" should be interpreted broadly to encompass any means configured to receive audio input data (i.e., audio data including speech) and process such data. For example, a listening device can be a conference speaker, i.e., a speaker or the like placed on a table for generating sound for one or more users around the table. A conference speaker may include a receiver means for receiving audio input data, one or more processors, and one or more memories, and is configured to process the audio input data into audio output data (i.e., audio data in which speech intelligibility has been improved compared to the received audio input data).

[0060] The listening device can be configured to receive audio input data via a data communication module. For example, the device can be a speakerphone configured to receive audio input data from an external device, such as a mobile phone connected via the listening device's data communication module. The device may also be equipped with a microphone arranged to convert incoming sound into audio input data.

[0061] A hearing device can also be a hearing aid, that is, one or two components worn by a user in one or both ears. Hearing aid components are known to include one or more microphones, a processor and memory for processing data received from the microphones, and one or more transducers for generating sound waves for the user of the hearing aid. In the case of two hearing aid components, these components can be configured to communicate with each other, thereby improving the auditory experience. Hearing aids can also be configured to communicate with external devices (such as mobile phones), in which case audio input data can be captured by the mobile phone and transmitted to the hearing aid. The mobile phone itself can also constitute a hearing device.

[0062] In this context, hearing aids should not be understood as devices used solely by people with hearing impairments, but rather as devices used by anyone interested in perceiving speech more clearly (i.e., improving speech intelligibility). When not used to provide audio output data, the listening device can be used for listening to music or similar purposes. In other words, the listening device can be an earpiece, headset, or other similar device configured to convert received audio input data into the audio output data described herein.

[0063] Listening devices can also be incorporated into devices that are not solely for hearing purposes. For example, a listening device can be a pair of smart glasses. In addition to converting audio input data into audio output data and providing the resulting sound via, for example, the lenses of smart glasses, as described herein, these glasses can also present visual information to the user by using the lenses as a heads-up display.

[0064] The listening device can also be a soundbar or other speaker used for listening to music or connected to a TV or monitor to provide sound to content displayed on the TV or monitor. As described herein, the conversion of incoming audio input data to audio output data can be performed either when the audio input data is provided alone or when the audio input data is provided together with visual data.

[0065] The hearing device can be configured to be worn by a user. The hearing device can be positioned at the user's ear, above the user's ear, in the user's ear canal, behind the user's ear, and / or in the user's outer ear; that is, the hearing device is configured to be worn inside, above, above, and / or at the user's ear. A user can wear two hearing devices, one in each ear. The two hearing devices can be connected (such as a binaural hearing aid system), such as via wireless and / or wired connection.

[0066] The listening device can be audible, such as a headset, headphones, earphones, earbuds, hearing aids, personal amplification products (PSAPs), over-the-counter (OTC) listening devices, hearing protection devices, one-size-fits-all listening devices, custom-made listening devices, or another headset. The listening device can be a speakerphone or a soundbar. The listening device can include both prescription and over-the-counter devices.

[0067] Listening devices can be embodied in various housing styles or shape elements. Some of these shape elements are earbuds, in-ear headphones, or ear-hook headphones. Those skilled in the art are aware of different types of listening devices and different options for positioning the listening device inside, above, or on the ear of the listener. The listening device (or a pair of listening devices) can be a custom adapter, a standard adapter, an open adapter, and / or a closed adapter.

[0068] The listening device may include one or more input transducers. The one or more input transducers may include one or more microphones. The one or more input transducers may include one or more vibration sensors configured to detect bone vibrations. The one or more input transducers may be configured to convert an acoustic signal into a first electrical input signal. The first electrical input signal may be an analog signal. The first electrical input signal may be a digital signal. The one or more input transducers may be coupled to one or more analog-to-digital converters configured to convert the analog first input signal into a digital first input signal.

[0069] The listening device may include one or more antennas configured for wireless communication. The one or more antennas may include an electrical antenna. The electrical antenna may be configured for wireless communication at a first frequency.

[0070] The first frequency can be above 800MHz, preferably between 900MHz and 6GHz.

[0071] The first frequency can be from 902MHz to 928MHz. The first frequency can be from 2.4GHz to 2.5GHz. The first frequency can be from 5.725GHz to 5.875GHz. One or more antennas may include a magnetic antenna. The magnetic antenna may include a magnetic core. The magnetic antenna may include a coil. The coil may be wound around the magnetic core. The magnetic antenna can be configured for wireless communication at a second frequency. The second frequency can be below 100MHz. The second frequency can be between 9MHz and 15MHz.

[0072] The listening device may include one or more wireless communication units. These units may include one or more wireless receivers, one or more wireless transmitters, one or more transmitter-receiver pairs, and / or one or more transceivers. At least one of the wireless communication units may be coupled to one or more antennas. The wireless communication units may be configured to convert wireless signals received by at least one of the antennas into a second electrical input signal. The listening device may be configured for wired / wireless audio communication, for example, enabling a user to listen to media such as music or radio, and / or enabling a user to make telephone calls.

[0073] The wireless signal may originate from one or more external sources and / or external devices, such as a spouse microphone device, a wireless audio transmitter, a smart computer, and / or a distributed microphone array associated with the wireless transmitter. The wireless input signal may originate from another listening device, such as as part of a binaural listening system, and / or from one or more accessory devices, such as a smartphone and / or a smartwatch.

[0074] The listening device may include a processing unit. The processing unit may be configured to process a first electrical input signal and / or a second electrical input signal. This processing may include compensating for a user's hearing loss, i.e., applying a frequency-related gain to the input signal based on the user's frequency-related hearing impairment. The processing may include performing feedback cancellation, beamforming, tinnitus reduction / masking, noise reduction, noise cancellation, speech recognition, bass adjustment, treble adjustment, and / or processing of user input. The processing unit may be a processor, integrated circuit, application program, functional module, etc. The processing unit may be implemented in a signal processing chip or a printed circuit board (PCB). The processing unit is configured to provide a first electrical output signal based on the processing of the first electrical input signal and / or the second electrical input signal. The processing unit may be configured to provide a second electrical output signal. The second electrical output signal may be based on the processing of the first electrical input signal and / or the second electrical input signal.

[0075] The listening device may include an output transducer. The output transducer may be coupled to a processing unit. The output transducer may be a loudspeaker. The output transducer may be configured to convert a first electrical output signal into an acoustic output signal. The output transducer may be coupled to the processing unit via a magnetic antenna.

[0076] In an embodiment, the wireless communication unit can be configured to convert the second electrical output signal into a wireless output signal. The wireless output signal may include synchronization data. The wireless communication unit can be configured to transmit the wireless output signal via at least one of one or more antennas.

[0077] The listening device may include a digital-to-analog converter configured to convert a first electrical output signal, a second electrical output signal, and / or a wireless output signal into an analog signal.

[0078] Listening devices may include vents. Vents are physical channels (such as channels or tubes) primarily positioned to provide pressure equalization across a housing placed in the ear, such as ITE listening devices, ITE units of BTE listening devices, CIC listening devices, RIE listening devices, RIC listening devices, MaRIE listening devices, or round-tipped earmolds. Vents may be pressure vents with a small cross-sectional area, preferably acoustically sealed. Vents may be acoustic vents configured to eliminate obstructions. Vents may be active vents, allowing them to be opened or closed during use of the listening device. Active vents may include valves.

[0079] The listening device may include a power source. The power source may include a battery providing a first voltage. The battery may be a rechargeable battery. The battery may be a replaceable battery. The power source may include a power management unit. The power management unit may be configured to convert the first voltage into a second voltage. The power source may include a charging coil. The charging coil may be provided by a magnetic antenna.

[0080] The listening device may include memory, including memory in volatile and non-volatile forms.

[0081] The listening device may include one or more antennas for radio frequency communication. The one or more antennas may be configured to operate in the ISM band. One of the one or more antennas may be an electric antenna. One of the one or more antennas may be a magnetic induction coil antenna. Magnetic induction, or near-field magnetic induction (NFMI), typically provides communication, including the transmission of voice, audio, and data, in the frequency range of 2 MHz to 15 MHz. At these frequencies, electromagnetic radiation propagates around the human head and body without significant attenuation in tissues.

[0082] The magnetic induction coil can be configured to operate at frequencies below 100 MHz during use, such as below 30 MHz, such as below 15 MHz. The magnetic induction coil can also be configured to operate within a frequency range between 1 MHz and 100 MHz, such as between 1 MHz and 15 MHz, such as between 1 MHz and 30 MHz, such as between 5 MHz and 30 MHz, such as between 5 MHz and 15 MHz, such as between 10 MHz and 11 MHz, such as between 10.2 MHz and 11 MHz. This frequency can also include a range from 2 MHz to 30 MHz, such as from 2 MHz to 10 MHz, such as from 2 MHz to 10 MHz, such as from 5 MHz to 10 MHz, such as from 5 MHz to 7 MHz.

[0083] The electric antenna can be configured to operate at frequencies of at least 400 MHz, such as at least 800 MHz, such as at least 1 GHz, such as between 1.5 GHz and 6 GHz, such as between 1.5 GHz and 3 GHz, such as at 2.4 GHz. The antenna can be optimized for operation at frequencies between 400 MHz and 6 GHz, such as between 400 MHz and 1 GHz, between 800 MHz and 1 GHz, between 800 MHz and 6 GHz, between 800 MHz and 3 GHz, etc. Therefore, the electric antenna can be configured to operate in the ISM band. The electric antenna can be any antenna capable of operating at these frequencies, and the electric antenna can be a resonant antenna, such as a monopole antenna, such as a dipole antenna, etc. The resonant antenna can have a length of λ / 4 ± 10% or any multiple thereof, where λ is the wavelength corresponding to the emitted electromagnetic field.

[0084] The present invention relates to various aspects, including listening devices and systems described above and below, and corresponding device portions, each aspect producing one or more advantages and benefits in conjunction with the first aspect, and each aspect having one or more embodiments corresponding to the embodiments described in conjunction with the first mentioned aspect and / or disclosed in the appended claims. Attached Figure Description

[0085] The above and other features and advantages will become apparent to those skilled in the art from the following detailed description of exemplary embodiments with reference to the accompanying drawings, wherein:

[0086] Figure 1 This schematically illustrates how audio input data is processed into audio output data using a listening device.

[0087] Figure 2 This is a flowchart illustrating a method for converting audio input data into audio output data. Detailed Implementation

[0088] Various embodiments are described below with reference to the accompanying drawings. Throughout the text, the same reference numerals denote the same elements. Therefore, the same elements will not be described in detail in the description of each drawing. It should also be noted that the drawings are intended only to facilitate the description of the embodiments. They are not intended as an exhaustive description of the claimed invention or a limitation on the scope of the claimed invention. Furthermore, the illustrated embodiments do not necessarily have all the aspects or advantages shown. The aspects or advantages described in connection with a particular embodiment are not necessarily limited to that embodiment and may be practiced in any other embodiment even if not so shown or so explicitly described.

[0089] Figure 1A listening device 100 is illustrated schematically by way of example. As shown, the listening device 100 may include a receiver RX device communicatively connected to a transmitter TX device via a communication channel. The listening device 100 may be a device in which both the transmitter TX device and the receiver RX device are integrated, but it may also be a device having only the receiver RX device.

[0090] For example, the listening device can be a hearing aid that includes both a microphone and an output transducer. In this example, the microphone can be linked to the TX device, and the output transducer can be linked to the RX device. According to another example, the listening device 100 can be a conference device equipped with a speaker. In this example, the conference device can include the RX device. The transmitter device TX in this example can be located in the telephone or computer of the user calling the conference device.

[0091] The transmitter device TX can be configured to receive audio data 102. This audio data 102 can be captured by one or more microphones included in the transmitter device TX, or from one or more external microphones. Once received, the audio data 102 can be fed to a transmitter-side universal noise reduction module (TX G-NR) 104, where audio input data 110 and time-frequency masking 108 are identified based on the audio data 102.

[0092] The TX G-NR 104 can include neural networks trained on a wide variety of sound types. For a specific example, the TX G-NR 102 can be set up following the teachings of M. Kolback's (2018) paper, "Single-Microphone Speech Enhancement and Separation Using Deep Learning." The TX G-NR 102 can be implemented using a feedforward architecture with an 1845-dimensional input layer and three hidden layers, each with 1024 hidden units and 64 output units (the same number as the gammatone filters). The enable function for the hidden units can be rectified linear units (ReLUs), and for the output units, a sigmoid function can be applied. The network can be designed for ideal ratio masking (IRM). As training data, Microsoft... TM The company provided the data as part of the DNS Challenge (see https: / / github.com / microsoft / DNS-Challenge). The IRM identified when evaluating audio data 102 can be used as a TF mask 108.

[0093] Audio input data 110 can be transmitted from the TX device to the RX device via a communication channel. Even if not shown, the audio input data 110 can be encoded before transmission. The audio input data 110 can be transmitted to the receiver-side universal noise reduction (RX G-NR) module 112 and dedicated noise reduction (S-NR) selector 127, which will be described in further detail below.

[0094] In parallel with audio input data 110, a TF mask 108 can be sent from the TX device to the RX device. In the RX device, the TF mask 108 can be received by a TF mask receiver 134. The TF mask receiver 134 can be communicatively connected to a TF mask detector 114. When the TF mask detector 114 detects that the TF mask 108 has been sent from the TX device, this information becomes available to the RX G-NR module 112, and it can be assumed that the audio input data 110 does not need to be processed by the RX G-NR module 112, which is a positive result that can improve computational power efficiency. On the other hand, if no TF mask 108 is detected from the TX device, the RX G-NR module 112 can be instructed to identify this situation by using the RX G-NR module 112. The advantage of this method is that two TX devices, configured to output the TF mask 108 and not output the TF mask 108, can be used with the RX device. Furthermore, as described above, when the TF mask is determined by the TX device, computational power can be saved by detecting that the TF mask has been identified.

[0095] Once the TF mask 108 is determined by the TX G-NR module 104 or the RX G-NR module 112, it is transmitted to the speech removal module 120. Since the TF mask 108 can be determined by two different modules, a switch 118 can be provided to ensure that the TF mask 108 is transmitted from one of the two modules to the speech removal module 120.

[0096] In addition to TF masking 108, audio input data 110 can also be transmitted to speech removal module 120. Background sound data 122 can be obtained by multiplying TF masking 108 with audio input data 110, more specifically by element-wise multiplication, or by combining the two datasets in any other way. In other words, TF masking 108 is not used to remove noise from audio input data 110, but rather to remove the speech components of audio input data 110. The remaining noise, referred to herein as background sound data 122, is then transmitted to acoustic scene classifier (ASC) module 124.

[0097] Although the example above uses TF masking 108 to provide background audio data 122, this is just one of several options. Typically, any auxiliary information extracted from the audio data 102 that can be used to distinguish between speech and non-speech components can be used.

[0098] Figure 1 Another option, not shown, is to use a neural network to determine the background sound data 122 without forming any auxiliary information, such as TF masking 108. If a neural network is used for this purpose, the background sound data 122 can be determined based on the audio input data 110 provided from the transmitter device TX. In the case of using a neural network as described above, the audio data 102 received by the transmitter device TX can be forwarded without modification, that is, the audio input data 110 can correspond to the audio data 102.

[0099] The ASC module 124 can link the background sound data 122 to one of multiple acoustic scenes. As an example, the background sound data 122 can be linked to three predetermined acoustic scenes associated with a transportation environment, an indoor environment, and an outdoor environment. However, the ASC module 124 can also be implemented such that it outputs a combination of predetermined acoustic scenes.

[0100] The ASC module 124 can be implemented in several different ways. One option is to use a deep convolutional layer network as the ASC module 124. The advantage of this is that such a network can efficiently extract hidden features from the input log-mel spectrogram. Using a convolutional layer network, sometimes called a convolutional neural network (CNN), may allow for a smaller number of parameters compared to other types of networks. This, in turn, brings hardware advantages. This is especially true for deep CNNs. Examples of networks proven beneficial for acoustic scene classification are further described in the paper “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition” (2019) by Q. Kong, Y. Cao, T. Iqbal, Y. Wang, and M. Plumbley. TMThe provided audio set was used to train the network (see link https: / / research.google.com / audioSet / ). This dataset includes 2.1 million clips from YouTube (5800 hours, 1.5+ TB, 527 categories). For fine-tuning training, the dataset provided by A. Mesaros, T. Heittola, and T. Virtanen on pages 9-13 of the 2018 Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), titled "A multi-device dataset for urban acoustic scene classification," was found to be helpful (40 hours, 30-second audio files, 12 European cities, two channels, recorded at 48Hz, 41.5 GB, 10 labels: airport, shopping mall, subway station, pedestrians on the street, public square, street transport, tram, bus, metro, park). In other words, by choosing the dataset above or a similar dataset as the training dataset, CNN will be suitable for the task of environmental classification; that is, CNN is becoming suitable for this specific sound classification rather than general sound classification.

[0101] An ASC module can be trained on pure ambient background noise (i.e., without speech). However, it can also be trained by considering audio data that includes both ambient background noise and speech, as well as ambient background noise after removing speech. The latter type of data can be obtained by using either the TX G-NR module 104 or the RX G-NR module 112 for speech removal. ASC modules can also be trained using combinations of all three, or combinations of two of them.

[0102] Once the background sound data 122 has been linked to the acoustic scene, acoustic scene data 126, including information about the identified acoustic scene, can be transmitted from the ASC module 124 to the dedicated noise reduction (S-NR) selector 127. Based on the acoustic scene data 126, S-NR modules 128A-C can be selected from multiple different S-NR modules 128A-C linked to different acoustic scenes.

[0103] S-NR modules 128A-C can have the same neural network architecture as G-NR modules (i.e., TX G-NR module 104 and RX G-NR module 112), but they can also have more complex structures. More complex structures can be achieved by having architectures based on multiple recurrent layers, such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) layers.

[0104] The S-NR module 128A-C can also be trained using data provided by Microsoft™ as part of the DNS Challenge (see https: / / github.com / Microsoft / DNS-Challenge). Furthermore, especially for environments that may include speech, datasets including audio input data can be considered. For example, the Voice Cloning Toolkit (VCTK) can also form part of the training data.

[0105] Once an S-NR module 128A-C matching the acoustic scene data 126 is selected, the audio input data 110 can be processed in the processing module 129 using the S-NR module 128A-C to generate audio output data 130. The difference between the audio output data 130 and the audio input data 110 is that speech intelligibility is improved.

[0106] When the TX device is part of a mobile phone carried by the user, the location data 132 connected to the TX device can be used as an additional input for selecting S-NR modules 128A-C. For example, if the location data indicates that the user is moving at a speed of 50 to 100 kilometers per hour, and if the ASC module 124 indicates that the acoustic scene is likely to be an indoor environment and a transportation environment, then considering the location data 132, a transportation environment may be selected. The location data 132 can be received by the location data receiver 136 in the RX device. Instead of being fed to the S-NR selector 127, the location data 132 can be provided to the ASC module 124, and thus form part of the acoustic scene recognition process.

[0107] Even if not explicitly shown, information about the identified acoustic scene can be provided to the user. For example, information about which acoustic scene is being used can be presented in a graphical user interface (GUI), such as a message indicating "Train audio settings are being used" in a software application on the user's mobile phone. If the acoustic scene classification is incorrect, the user can choose to override the selection made by the ASC module 124 and make a manual selection. Another possibility is to proactively raise a question to the user when the acoustic scene classification cannot become reliable (i.e., when the background sound data 122 does not directly match any preset environment).

[0108] Figure 2 This is a flowchart illustrating the steps of a method 200 for converting audio input data into audio output data, i.e., how audio data is processed to improve speech clarity.

[0109] In the first step 202, audio input data 110 can be received. Once received, background sound data 122 (sometimes referred to as noise, sometimes as ambient sound data) can be provided in the second step 204 by separating speech components from the audio input data 110. Subsequently, in the third step 206, acoustic scene data 126 can be determined. Based on the acoustic scene data 126, in the fourth step 208, S-NR modules 128A-C can be selected. Subsequently, in the fifth step 210, speech enhancement data 130 can be generated by using S-NR modules 128A-C.

[0110] Optionally, in the sixth step 212, the RX device may receive the TF mask 108 or other side information. After receiving it, in the seventh step 214, the TF mask 108 may be provided to the speech removal module 120 so that background sound data 122 is generated by combining the audio input data 110 with the TF mask 108.

[0111] Optionally, in step 216, it can be determined whether TF mask 108 is received from the TX device. If TF mask 108 is received, the RX G-NR module 112 can be disabled, that is, it is not used to identify TF mask 108. On the other hand, if the RX device does not receive TF mask, in step 218, the RX G-NR module 112 can be enabled, and TF mask can be provided by the RX G-NR module 112.

[0112] As described above, if it is determined that the TF mask 108 provided by the TX device does not exist, step 218 can be performed. Alternatively, if the TX device is not provided with the function of determining the TF mask 108, the RX G-NR module can be configured to perform step 218 instead of step 216. Once step 218 is performed, in step 220, the TF mask 108 is provided to the speech removal module to generate background sound data 122.

[0113] Furthermore, in step 11 222, location data 132 can be received, and once the location data is received, it can be used as input to the decision-making process for selecting S-NR modules 128A-C.

[0114] Although specific features have been shown and described, it should be understood that they are not intended to limit the claimed invention, and it will be apparent to those skilled in the art that various changes and modifications can be made without departing from the scope of the claimed invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The claimed invention is intended to cover all alternatives, modifications, and equivalents.

[0115] Reference List

[0116] 100- Listening Device

[0117] 102-Audio Data

[0118] 104-Transmitter-Side Universal Noise Reduction (TX G-NR) Module

[0119] 108-Time-Frequency (TF) Masking

[0120] 110 - Audio Input Data

[0121] 112-Receiver-side Universal Noise Cancellation (RX G-NR) Module

[0122] 114-TF Mask Selector

[0123] 118-Switch

[0124] 120-Voice Removal Module

[0125] 122-Background sound data

[0126] 124-ASC module

[0127] 126-Acoustic Scene Data

[0128] 127-Dedicated Noise Reduction (S-NR) Selector

[0129] 128A-C Dedicated Noise Reduction (S-NR) Module

[0130] 129-Processing Module

[0131] 130-Speech Enhancement Data

[0132] 132-Location Data

[0133] 134-TF Masked Receiver

[0134] 136-Location Data Receiver

[0135] 200-220- Methods for converting audio input data into audio output data and the different steps associated with the method.

Claims

1. A computer-implemented method (200) for converting audio input data (110) into audio output data (130), the method (200) comprising: Receive (202) audio input data (110). Background sound data (122) is provided (204) by separating speech components from the audio input data (110) using a speech removal module (120). By using the Acoustic Scene Classifier (ASC) module (124), acoustic scene data (126) linked to the acoustic scene (AS) that matches the background sound data (122) is determined (206). Based on the acoustic scene data (126), select (208) a dedicated noise reduction (S-NR) module (128A-C), and The audio input data (110) is processed (210) by the dedicated noise reduction (S-NR) module (128A-C) to generate the audio output data (130). The receiver-side universal noise reduction (RX G-NR) module (112) and TF masking detector (114) are arranged in the receiver (RX) device, and the method further includes: Determine (216) whether TF masking (108) is received from the transmitter (TX) device. If the TF masking (108) is received via the RX device, the RX G-NR module (112) is disabled, or If the TF mask (108) is not received through the RX device, the RX G-NR module (112) is enabled so that the TF mask (108) is provided through the RX G-NR module (112).

2. The method of claim 1, wherein, The S-NR module (128A-C) is a neural network, and the dedicated noise reduction (S-NR) module (128A-C) is selected from a fixed set of pre-trained neural networks, each of which is designed for a sound environment with specific characteristics.

3. The method according to claim 2, wherein, The fixed set includes at least three modules, including a module for the transportation environment, a module for the outdoor environment, and a module for the indoor environment.

4. The method according to claim 1, wherein, The step of receiving (202) the audio input data (110) is performed at the RX device, and the audio input data (110) is sent from the TX device. The method further includes: The TF mask (108) from the TX device is received (212) at the RX device. The speech removal module (120) arranged in the RX device is provided with a TF mask (108) (214) such that the background sound data (122) is generated by combining the audio input data (110) with the TF mask (108).

5. The method according to claim 4, wherein, The step of providing the background sound data (122) is performed by multiplying the audio input data (110) with the TF mask (108).

6. The method according to claim 4 or 5, wherein, The audio input data (110) and the TF mask (108) are received in parallel through the RX device.

7. The method according to claim 4 or 5, wherein, Each of the S-NR modules (128A-C) is more complex than the receiver-side universal noise reduction (RX G-NR) module (112), which is configured to identify the TF mask (108) based on the audio input data (110) such that the computational power associated with each of the S-NR modules (128A-C) is greater than the computational power associated with the RX G-NR module (104).

8. The method according to any one of claims 1 to 5, further comprising: The TF mask (108) is identified (218) by using the RX G-NR module (112) on the audio input data (110), and The TF mask (108) is provided (220) to the speech removal module (120) so that the background sound data (122) is generated by combining the audio input data (110) with the TF mask (108).

9. The method according to any one of claims 1 to 5, further comprising: Receive (222) location data (132). The step of selecting (208) S-NR module (128A-C) is based on the combination of the acoustic scene data (126) and the location data (132).

10. A listening device (100), comprising: A receiver (RX) device is configured to receive audio input data (110) from a transmitter (TX) device. The RX device also includes: The speech removal module (120) is configured to provide background sound data (122) by separating speech components from the audio input data (110). The acoustic scene classifier (ASC) module (124) is configured to determine acoustic scene data (126) that is linked to the acoustic scene (AS) that matches the background sound data (122). A dedicated noise reduction (S-NR) selector (127) is configured to select a dedicated noise reduction (S-NR) module (128A-C) based on the acoustic scene data (126). The processing module (129) is configured to process the audio input data (110) by using the dedicated noise reduction (S-NR) module (128A-C) to generate audio output data (130). A receiver universal noise reduction (RX G-NR) module (112) is configured to provide TF masking (108) based on the audio input data (110), and The TF mask detector (114) is configured to identify whether the TF mask (108) is sent from the TX device, and to disable the RX G-NR module (112) if the TF mask (108) is sent from the TX device, or to enable the RX G-NR module (112) if the TF mask (108) is not sent from the TX device, so that the TF mask (108) is provided through the RX G-NR module (112).

11. The listening device (100) according to claim 10, wherein, The RX device further includes: The TF masking receiver (134) is configured to receive TF masking (108) from the TX device. The speech removal module (120) is configured to remove the speech component from the audio input data (110) by combining the audio input data (110) and the TF mask (108).

12. The listening device (100) according to claim 10, wherein, The RX G-NR module (112) is configured to provide a TF mask (108) based on the audio input data (110) and to provide the TF mask (108) to the speech removal module (120) such that the background sound data (122) is generated by combining the audio input data (110) with the TF mask (108).

13. The listening device (100) according to claim 10 or 11, wherein, The RX device further includes: The location data receiver (136) is configured to receive location data (132) from the TX device. The S-NR selector (127) is configured to select an S-NR module (128A-C) based on a combination of the acoustic scene data (126) and the location data (132).

Images