Noise reduction using machine learning

Abstract

A method of noise reduction includes using a neural network to control a Wiener filter. The gain estimated by the neural network is combined with the gain produced by the Wiener filter. In this way, the noise reduction system provides improved results compared to using only the neural network.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to European Patent Application No. 20206921.7, filed November 11, 2020; U.S. Provisional Patent Application No. 63 / 110,114, filed November 5, 2020; U.S. Provisional Patent Application No. 63 / 068,227, filed August 20, 2020; and International Patent Application No. PCT / CN2020 / 106270, filed July 31, 2020, all of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to audio processing, and more specifically to noise reduction. Background Technology

[0004] Unless otherwise stated herein, the methods described in this section are not prior art to the claims of this application and are not acknowledged as prior art by virtue of being included in this section.

[0005] Implementing noise reduction in mobile devices is challenging. Mobile devices can capture both static and non-static noise in a variety of use cases, including voice communication, user-generated content development, and more. The limitations of mobile devices in terms of power consumption and processing power further complicate the development of effective noise reduction processes when implemented on mobile devices. Summary of the Invention

[0006] In view of the above, there is a need to develop a noise reduction system that works well in mobile devices.

[0007] According to one embodiment, a computer-implemented audio processing method includes generating a first frequency band gain and speech activity detection values ​​of an audio signal using a machine learning model. The method further includes generating a background noise estimate based on the first frequency band gain and the speech activity detection values. The method also includes generating a second frequency band gain by processing the audio signal using a Wiener filter controlled by the background noise estimate. The method further includes generating a combined gain by combining the first and second frequency band gains. The method also includes generating a modified audio signal by modifying the audio signal using the combined gain.

[0008] According to another embodiment, an apparatus includes a processor and a memory. The processor is configured to control the apparatus to implement one or more of the methods described herein. The apparatus may additionally include details similar to those of the methods described herein.

[0009] According to another embodiment, a non-transitory computer readable medium stores a computer program which, when executed by a processor, controls an apparatus to perform processing comprising one or more of the methods described herein.

[0010] The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a block diagram of a noise reduction system 100.

[0012] Figure 2 shows a block diagram of an example system 200 suitable for implementing example embodiments of the present disclosure.

[0013] Figure 3 is a flowchart of a method 300 of audio processing. DETAILED DESCRIPTION

[0014] Techniques related to noise reduction are described herein. In the following description, for the purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, to one skilled in the art that the present disclosure can be practiced without some or all of these details, or with variations of the details described below, and that the present disclosure can include modifications and equivalents of the features and concepts described herein.

[0015] In the following description, various methods, processes and procedures are described. While specific steps can be described in a certain order, such order is primarily for convenience and clarity. Specific steps can be repeated more than once, can occur before or after other steps, even if those steps are described elsewhere in another order, and can occur in parallel with other steps. Only when a first step must be completed before a second step can be started does the second step occur after the first step. This will be specifically noted when it is not clear from the context.

[0016] In this document, the terms“and”,“or” and“and / or” are used. Such terms are to be read as being inclusive. For example,“A and B” can mean at least the following:“both A and B”,“at least A and at least B”. As another example,“A or B” can mean at least the following:“at least A”,“at least B”,“both A and B”,“at least both A and B”. As another example,“A and / or B” can mean at least the following:“A and B”,“A or B”. When intended to be exclusive, it will be specifically noted (e.g.,“A or B”,“at most one of A and B”).

[0017] This document describes the various processing functions associated with structures such as blocks, elements, components, and circuits. Typically, these structures can be implemented by a processor controlled by one or more computer programs.

[0018] Figure 1 This is a block diagram of a noise reduction system 100. The noise reduction system 100 can be used in mobile devices such as mobile phones, cameras with microphones, etc. (e.g., see...). Figure 2 The noise reduction system 100 is implemented in a processor, for example, controlled by one or more computer programs. The noise reduction system 100 includes a windowing block 102, a transform block 104, a frequency band feature analysis block 106, a neural network 108, a Wiener filter 110, a gain combination block 112, a frequency band gain to bin gain block 114, a signal modification block 116, an inverse transform block 118, and an inverse windowing block 120. The noise reduction system 100 may include other components (for brevity) not described in detail.

[0019] Windowing block 102 receives audio signal 150, performs windowing on audio signal 150, and generates audio frame 152. Audio signal 150 can be captured by a microphone of a mobile device implementing noise reduction system 100. Typically, audio signal 150 is a time-domain signal comprising a sequence of audio samples. For example, audio signal 150 can be captured at a sampling rate of 48 kHz, where each sample is quantized at a bit rate of 16 bits. Other example sampling rates can include 44.1 kHz, 96 kHz, 192 kHz, etc., and other bit rates can include 24 bits, 32 bits, etc.

[0020] Typically, windowing block 102 applies overlapping windows to samples of audio signal 150 to generate audio frame 152. Windowing block 102 can implement various forms of windowing, including rectangular windows, triangular windows, trapezoidal windows, sine windows, etc.

[0021] Transform block 104 receives audio frame 152, performs a transform on audio frame 152, and generates transform feature 154. This transform can be a frequency domain transform, and transform feature 154 can include bin features and fundamental frequency parameters for each audio frame. (Transform feature 154 can also be referred to as bin feature 154.) The fundamental frequency parameter can include the speech fundamental frequency referred to as F0. Transform block 104 can implement various transforms, including Fourier transform (e.g., Fast Fourier Transform (FFT)), Quadrature Mirror Filter (QMF) domain transform, etc. For example, transform block 104 can implement an FFT with a 960-point analysis window and a 480-point frame shift; alternatively, it can implement a 1024-point analysis window and a 512-point frame shift. The number of bins in transform feature 154 is typically related to the number of points analyzed in the transform; for example, a 960-point FFT produces 481 bins.

[0022] The transform block 104 can implement various processes to determine the fundamental frequency parameter for each audio frame. For example, when the transform is an FFT, the transform block 104 can extract the fundamental frequency parameter from the FFT parameters. As another example, the transform block 104 can extract the fundamental frequency parameter based on the autocorrelation of the time-domain signal (e.g., the audio frame 152).

[0023] The band feature analysis block 106 receives the transformed features 154, performs band analysis on the transformed features 154, and generates band features 156. The band features 156 can be generated according to various scales, including a Mel scale, a Bark scale, and the like. When different scales are used, the number of bands in the band features 156 can be different, e.g., 24 bands for a Bark scale, 80 bands for a Mel scale, and the like. The band feature analysis block 106 can combine the band features 156 with the fundamental frequency parameter (e.g., F0).

[0024] The band feature analysis block 106 can use rectangular bands. The band feature analysis block 106 can also use triangular bands, where the peak response is located at the border between bands.

[0025] The band features 156 can be band energies, such as Mel band energies, Bark band energies, and the like. The band feature analysis block 106 can compute the log values of the Mel band energies and the Bark band energies. The band feature analysis block 106 can apply a discrete cosine transform (DCT) conversion of the band energies to generate new band features that are less correlated than the original band features. For example, the band feature analysis block 106 can generate the band features 156 as Mel frequency cepstral coefficients (MFCCs), Bark frequency cepstral coefficients (BFCCs), and the like.

[0026] The band feature analysis block 106 can perform smoothing of the current frame and the previous frame according to a smoothing value. The band feature analysis block 106 can also perform difference analysis by computing a first-order difference and a second-order difference between the current frame and the previous frame.

[0027] The band feature analysis block 106 can compute a band harmonic feature that indicates how much of the current band is composed of a periodic signal. For example, the band feature analysis block 106 can compute the band harmonic feature based on the FFT frequency bins of the current frame. As another example, the band feature analysis block 106 can compute the band harmonic feature based on the correlation between the current frame and the previous frame.

[0028] In general, the band features 156 are fewer in number than the bin features 154, thus reducing the dimensionality of the data input to the neural network 108. For example, the bin features can be on the order of 513 bins or 481 bins, and the band features 156 can be on the order of 24 bands or 80 bands.

[0029] The neural network 108 receives the band features 156, processes the band features 156 according to a model, and generates gains 158 and voice activity decisions (VADs) 160. The gains 158 can also be referred to as D-gains, for example, to indicate that the gains are the output of the neural network. The model has been trained offline; training the model, including preparing a training data set, will be discussed in subsequent sections.

[0030] The neural network 108 uses the model to estimate the gain and voice activity for each band based on the band features 156 (e.g., including the fundamental frequency F0) and outputs the gains 158 and VADs 160. The neural network 108 can be a fully connected neural network (FCNN), a recurrent neural network (RNN), a convolutional neural network (CNN), another type of machine learning system, or the like, or a combination thereof.

[0031] The noise reduction system 100 can apply smoothing or limiting to the D-gain output of the neural network 108. For example, the noise reduction system 100 can apply average smoothing or median filtering to the gains 158 along a time axis, a frequency axis, or the like. As another example, the noise reduction system 100 can apply limiting to the gains 158, where the maximum gain is 1.0 and the minimum gain is different for different bands. In one implementation, the noise reduction system 100 sets a minimum gain of 0.1 (e.g., -20 dB) for the lowest 4 bands and a minimum gain of 0.18 (e.g., -15 dB) for the middle bands. Setting the minimum gain mitigates discontinuities in the D-gains. The minimum gain values can be adjusted as needed; for example, -12 dB, -15 dB, -18 dB, -20 dB, or the like, minimum gains can be set for various bands.

[0032] The Wiener filter 110 receives the band characteristics 156, the gains 158, and the VAD 160, performs Wiener filtering, and generates gains 162. The gains 162 can also be referred to as W-gains, for example, to indicate that the gains are the output of the Wiener filter. Generally, the Wiener filter 110 estimates the background noise in each band of the input signal 150 according to the band characteristics 156. (The background noise can also be referred to as stationary noise.) The Wiener filter 110 uses the gains 158 and the VAD 160 estimated by the neural network to control its filtering process. In one implementation, for a given input frame (with corresponding band characteristics 156) that has no speech activity (e.g., the VAD 160 is less than 0.5), the Wiener filter 110 checks the band gains for the given input frame (according to the gains 158 (D-gains)). For bands with D-gains less than 0.5, the Wiener filter 110 considers these bands as noise frames, and smooths the band energies of these frames to obtain an estimate of the background noise.

[0033] The Wiener filter 110 can also track the average number of frames used to compute the band energy for each band to obtain a noise estimate. When this average number for a given band is greater than a threshold number of frames, the Wiener filter 110 is applied to compute the Wiener band gain for the given band. If this average number for a given band is less than the threshold number of frames, the Wiener band gain for the given band is 1.0. The Wiener band gains for each band are output as the gains 162, also referred to as the Wiener gains (or W-gains).

[0034] In practice, the Wiener filter 110 estimates the background noise in each band based on the signal history (e.g., the number of frames of the input signal 150). The threshold number of frames gives the Wiener filter 110 enough frames to have a reliable estimate of the background noise. In one implementation, the threshold number of frames is 50. When a frame is 10 ms, this corresponds to 0.5 seconds of the input signal 150. When the number of frames is less than the threshold, the Wiener filter 110 is effectively bypassed (e.g., the W-gain is 1.0).

[0035] The noise reduction system 100 can apply a limit to the W-gain output of the Wiener filter 110, where the maximum gain is 1.0 and the minimum gain is different for different bands. In one implementation, the noise reduction system 100 sets a gain of 0.1 (e.g., -20 dB) as the minimum gain for the lowest 4 bands, and sets a gain of 0.18 (e.g., -15 dB) as the minimum gain for the middle bands. Setting the minimum gain mitigates discontinuities in the W-gain. The minimum gain values can be adjusted as needed; for example, -12 dB, -15 dB, -18 dB, -20 dB, etc. minimum gains can be set for various bands.

[0036] The gain combination block 112 receives the gain 158 (D gain) and the gain 162 (W gain), combines the gains, and generates a gain 164. The gain 164 can also be referred to as a band gain, a combined band gain, or a C gain, for example, to indicate that the gain is a combination of the D gain and the W gain. As an example, the gain combination block 112 can multiply the D gain and the W gain to generate the C gain on a per-band basis.

[0037] The noise reduction system 100 can apply a limit to the C gain output of the gain combination block 112, where the maximum gain is 1.0 and the minimum gain is different for different bands. In one implementation, the noise reduction system 100 sets a gain of 0.1 (e.g., -20 dB) as the minimum gain for the lowest 4 bands and sets a gain of 0.18 (e.g., -15 dB) as the minimum gain for the middle bands. Setting the minimum gain alleviates discontinuities in the C gain. The minimum gain values can be adjusted as needed; for example, -12 dB, -15 dB, -18 dB, -20 dB, etc. minimum gains can be set for various bands.

[0038] The band gain to bin gain block 114 receives the gain 164, converts the band gain to a bin gain, and generates a gain 166 (also referred to as a bin gain). In effect, the band gain to bin gain block 114 performs processing that is the inverse of the processing performed by the band characteristics analysis block 106 in order to convert the gain 164 from a band gain to a bin gain. For example, if the band characteristics analysis block 106 processes 1024-point FFT bins into 24 Bark-scaled bands, the band gain to bin gain block 114 converts the 24 Bark-scaled bands of the gain 164 into 1024 FFT bins of the gain 166.

[0039] The band gain to bin gain block 114 can implement various techniques to convert the band gain to the bin gain. For example, the band gain to bin gain block 114 can use interpolation, such as linear interpolation.

[0040] The signal modification block 116 receives the transformed features 154 (including the bin features and the fundamental frequency F0) and the gain 166, modifies the transformed features 154 according to the gain 166, and generates modified transformed features 168 (which include modified bin features and the fundamental frequency F0). (The modified transformed features 168 can also be referred to as modified bin features 168.) The signal modification block 116 can modify the amplitude spectrum of the bin features 154 based on the gain 166. In one implementation, the signal modification block 116 will leave the phase spectrum of the bin features 154 unchanged when generating the modified bin features 168. In another implementation, the signal modification block 116 will adjust the phase spectrum of the bin features 154 when generating the modified bin features 168, e.g., by performing an estimation based on the modified bin features 168. As an example, the signal modification block 116 can use a short-time Fourier transform to adjust the phase spectrum, e.g., by implementing a Griffin-Lim process.

[0041] The inverse transform block 118 receives the modified transformed features 168, performs an inverse transform on the modified transformed features 168, and generates audio frames 170. Typically, the inverse transform performed is the inverse of the transform performed by the transform block 104. For example, the inverse transform block 118 can implement an inverse Fourier transform (e.g., an inverse FFT), an inverse QMF transform, etc.

[0042] The inverse windowing block 120 receives the audio frames 170, performs an inverse windowing on the audio frames 170, and generates audio signals 172. Typically, the inverse windowing performed is the inverse of the windowing performed by the windowing block 102. For example, the inverse windowing block 120 can perform overlap-add on the audio frames 170 to generate the audio signals 172.

[0043] As a result, the combination of using the output of the neural network 108 to control the Wiener filter 110 can provide better results than using a neural network alone to perform noise reduction because many neural networks operate using only short-term memory.

[0044] Figure 2 A block diagram of an example system 200 suitable for implementing the example embodiments of the present disclosure is shown. The system 200 includes one or more server computers or any client devices. The system 200 includes any consumer device, including but not limited to a smart phone, a media player, a tablet, a laptop, a wearable computer, a vehicle computer, a game console, a surround system, a kiosk, etc.

[0045] As shown, the system 200 includes a central processing unit (CPU) 201 that is capable of executing various processes according to programs stored in, for example, a read only memory (ROM) 202 or loaded from, for example, a storage unit 208 to a random access memory (RAM) 203. In the RAM 203, data required when the CPU 201 executes various processes is also stored as necessary. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0046] The following components are connected to the I / O interface 205: an input unit 206 that can include a keyboard, a mouse, a touch screen, a motion sensor, a camera, and the like; an output unit 207 that can include a display such as a liquid crystal display (LCD) and one or more speakers; a storage unit 208 that includes a hard disk or other suitable storage device; and a communication unit 209 that includes a network interface card such as a wired or wireless network card. The communication unit 209 can also communicate with wireless input and output components such as a wireless microphone, a wireless earbud, a wireless speaker, and the like.

[0047] In some implementations, the input unit 206 includes one or more microphones at different locations depending on the host device, enabling the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0048] In some implementations, the output unit 207 includes a system with various numbers of speakers. As Figure 2 shown, the output unit 207 can present audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats) depending on the capabilities of the host device.

[0049] The communication unit 209 is configured to communicate with other devices (e.g., via a network). A driver 210 is also connected to the I / O interface 205 as necessary. A removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a flash memory drive, or other suitable removable medium is mounted on the drive 210, enabling a computer program read from the removable medium to be installed into the storage unit 208. Those skilled in the art will appreciate that, although the system 200 is described as including the components mentioned above, some of these components can be added, removed, and / or replaced in actual applications, and all such modifications or changes fall within the scope of the present disclosure.

[0050] For example, the system 200 can implement the noise reduction system 100 (see Figure 1The ROM 202, the RAM 203, the storage unit 208, and the like can store models used by the neural network 108. A microphone connected to the input unit 206 can capture the audio signal 150, and a speaker connected to the output unit 207 can output sound corresponding to the audio signal 172.

[0051] Figure 3 is a flowchart of a method 300 of audio processing. The method 300 can be implemented by a device (e.g., the system 200 of Figure 2 as controlled by execution of one or more computer programs.

[0052] At 302, a first band gain and a voice activity detection value for an audio signal are generated using a machine learning model. For example, the CPU 201 can implement the neural network 108 to generate the gain 158 and the VAD 160 by processing the band features 156 according to the model (see Figure 1 ).

[0053] At 304, a background noise estimate is generated based on the first band gain and the voice activity detection value. For example, as part of operating the Wiener filter 110, the CPU 201 can generate the background noise estimate based on the gain 158 and the VAD 160.

[0054] At 306, a second band gain is generated by processing the audio signal using a Wiener filter controlled by the background noise estimate. For example, the CPU 201 can implement the Wiener filter 110 to generate the gain 162 by processing the band features 156 as controlled by the background noise estimate (see 304). For example, when the number of noise frames exceeds a threshold (e.g., 50 noise frames) for a particular band, the Wiener filter generates the second band gain for that particular band.

[0055] At 308, a combined gain is generated by combining the first band gain and the second band gain. For example, the CPU 201 can implement the gain combination block 112 to generate the gain 164 by combining the gain 158 (from the neural network 108) and the gain 162 (from the Wiener filter 110). The first band gain and the second band gain can be combined by multiplication. The first band gain and the second band gain can be combined by selecting the maximum value of the first band gain and the second band gain for each band. A limit can be applied to the combined gain. The first band gain and the second band gain can be combined by multiplication or by selecting the maximum value for each band, and a limit can be applied to the combined gain.

[0056] At 310, a modified audio signal is generated by modifying the audio signal using the combined gains. For example, the CPU 201 can implement the signal modification block 116 to generate the modified bin features 168 by modifying the bin features 154 using the gains 166.

[0057] The method 300 can include other steps similar to those described above with respect to the noise reduction system 100. A non-exhaustive discussion of example steps includes the following. As part of generating the input to the neural network 108, a windowing step can be performed on the audio signal (see the windowing block 102). As part of generating the input to the neural network 108, a transform step can be performed on the audio signal (see the transform block 104) to convert time domain information to frequency domain information. A bin-to-band conversion step can be performed on the audio signal (see the band feature analysis block 106) to reduce the dimensionality of the input to the neural network 108. A band-to-bin conversion step can be performed (see the band gain to bin gain block 114) to convert the band gains (e.g., the gains 164) to bin gains (e.g., the gains 166). An inverse transform step can be performed (see the inverse transform block 118) to convert the modified bin features 168 from frequency domain information to time domain information (e.g., the audio frames 170). As an inverse step to the windowing step, an inverse windowing step can be performed (see the inverse windowing block 120) to reconstruct the audio signal 172.

[0058] Model creation

[0059] As described above, the model used by the neural network 108 (see Figure 1 ) can be trained offline and then stored and used by the noise reduction system 100. For example, a computer system can train the model, e.g., by executing one or more computer programs implementing a model training system. Part of training the model includes preparing training data to generate input features and target features. The input features can be computed by band feature computation of noise data (X). The target features consist of ideal band gains and VAD decisions.

[0060] The noise data (X) can be generated by combining clean speech (S) and noise data (N).

[0061] X = S + N

[0062] The VAD decisions can be based on analysis of the clean speech S. In one implementation, the VAD decisions are determined by an absolute threshold of the energy of the current frame. Other VAD methods can be used in other implementations. For example, the VAD can be manually labeled.

[0063] The ideal band gains g are computed by:

[0064]

[0065] In the above equation, E s (b) is the energy of the frequency band b of the clean speech, and E x (b) is the energy of the frequency band b of the noisy speech.

[0066] To make the model robust to different use cases, the model training system can perform data augmentation on the training data. Given a training data with S i and N i input speech files, the model training system will change S i and N i before mixing the noise data. The data augmentation includes three general steps.

[0067] The first step is to control the amplitude of the clean speech. One common problem with noise reduction models is that they suppress low volume speech. Thus, the model training system performs data augmentation by preparing training data that contains speech with various amplitudes.

[0068] The model training system sets a random target average amplitude ranging from -45 dB to 0 dB (e.g., -45, -40, -35, -30, -25, -20, -15, -10, -5, 0). The model training system modifies the input speech file by a value a to match the target average amplitude.

[0069] S m = a * S i

[0070] The second step is to control the signal-to-noise ratio (SNR). For each combination of speech and noise files, the model training system will set a random target SNR. In one implementation, the target SNR is randomly selected from a set of SNRs [-5, -3, 0, 3, 5, 10, 15, 18, 20, 30] with equal probability. The model training system then modifies the input noise file by a value b to match the SNR between S m and N m to the target SNR:

[0071] N m = b * N i

[0072] The third step is to limit the mixed data. The model training system first computes the mixed signal X ｍ by:

[0073] X ｍ = (S m + N ｍ )

[0074] In the case of clipping (e.g., when quantizing X ｍThe maximum absolute value of X (when saved as a.wav file), denoted A ｍ is computed by the model training system max .

[0075] The modification rate c can then be computed by:

[0076] c = 32767 / A max

[0077] In the above equation, the value 32,767 comes from 16-bit quantization; this value can be adjusted as needed for other bit quantization precisions.

[0078] Then:

[0079] S = c * S m

[0080] N = c * N m

[0081] S and N will be mixed as noisy speech X:

[0082] X = S + N

[0083] The computation of the average amplitude and SNR can be performed according to various procedures as needed. The model training system can use a minimum threshold to remove silent segments before computing the average amplitude.

[0084] In this way, by using various target average amplitudes and target SNRs to adjust segments of the training data, data augmentation is used to increase the diversity of the training data. For example, using 10 variations of the target average amplitude and 10 variations of the target SNR gives 100 variations of a single training data segment. Data augmentation does not require increasing the size of the training data. If the training data is 100 hours before data augmentation, it is not necessary to train the model using a full set of augmented training data of 10,000 hours; the augmented training data set can be limited to a smaller size, e.g., 100 hours. More importantly, data augmentation will increase the variability of the amplitudes and SNRs in the training data.

[0085] Implementation details

[0086] Embodiments can be implemented in hardware, executable modules stored on a computer readable medium, or a combination of both (e.g., programmable logic arrays). The steps implemented by embodiments need not be inherently related to any particular computer or other apparatus, although they can be related in some embodiments. In particular, various general purpose machines can be used with computer programs written in accordance with the teachings herein, or it can be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments can be implemented in one or more computer programs executing on one or more programmable computer systems each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices, in known fashion.

[0087] Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer system. The system and methods can be implemented in a computer program product tangibly embodied in a machine-readable storage medium (e.g., a storage device or memory element) for execution by, or to control the operation of, a computer system. The system and methods can also be implemented in a computer system having a computer program tangibly embodied in a machine-readable storage medium (e.g., a storage device or memory element) for execution by, or to control the operation of, the computer system.

[0088] The above description illustrates various embodiments of the disclosure and examples of how aspects of the disclosure can be implemented. The above examples and embodiments should not be considered to be only ones, and are presented to illustrate the flexibility and advantages of the disclosure defined by the appended claims. Other arrangements, embodiments, implementations, and equivalents, based on the above disclosure and appended claims, will be apparent to those skilled in the art and can be employed without departing from the spirit and scope of the disclosure defined by the claims.

[0089] Various aspects of the disclosure can be appreciated from the following enumerated example embodiments (EEEs):

[0090] EEE 1. A computer-implemented method of audio processing, the method comprising:

[0091] generating, using a machine learning model, a first band gain and a voice activity detection value for an audio signal;

[0092] generating a background noise estimate based on the first band gain and the voice activity detection value;

[0093] generating a second band gain by processing the audio signal using a Wiener filter controlled by the background noise estimate;

[0094] generating a combined gain by combining the first band gain and the second band gain; and

[0095] generating a modified audio signal by modifying the audio signal using the combined gain.

[0096] EEE 2. The method of EEE 1, wherein the machine learning model is generated using data augmentation to increase diversity of training data.

[0097] EEE 3. The method of any of EEEs 1-2, wherein generating the first band gain and the voice activity detection value is performed using one of a fully connected neural network, a recurrent neural network, and a convolutional neural network.

[0098] EEE 4. The method of any of EEEs 1-3, wherein generating the first band gain includes limiting the first band gain using at least two different limits for at least two different bands.

[0099] EEE 5. The method of any of EEEs 1-4, wherein generating the background noise estimate is based on a number of noise frames exceeding a threshold for a particular band.

[0100] EEE 6. The method of any of EEEs 1-5, wherein generating the second band gain includes using a Wiener filter based on a steady state noise level for a particular band.

[0101] EEE 7. The method of any of EEEs 1-6, wherein generating the second band gain includes limiting the second band gain using at least two different limits for at least two different bands.

[0102] EEE 8. The method of any of EEEs 1-7, wherein generating the combined gain includes:

[0103] multiplying the first band gain and the second band gain; and

[0104] limiting the combined band gain using at least two different limits for at least two different bands.

[0105] EEE 9. The method of any of EEEs 1-8, wherein generating the modified audio signal includes modifying a magnitude spectrum of the audio signal using the combined band gain.

[0106] EEE 10. The method of any of EEEs 1-9, further comprising:

[0107] applying an overlap window to the input audio signal to generate a plurality of frames, wherein the audio signal corresponds to the plurality of frames.

[0108] EEE 11. The method of any of EEEs 1-10, further comprising:

[0109] performing a spectral analysis on the audio signal to generate a plurality of bin features and a fundamental frequency of the audio signal,

[0110] wherein the first band gains and the voice activity detection value are based on the plurality of bin features and the fundamental frequency.

[0111] EEE 12. The method of EEE 11, further comprising:

[0112] generating a plurality of band features based on the plurality of bin features, wherein the plurality of band features are generated using one of mel frequency cepstral coefficients and bark frequency cepstral coefficients,

[0113] wherein the first band gains and the voice activity detection value are based on the plurality of band features and the fundamental frequency.

[0114] EEE 13. The method of any of EEEs 1-12, wherein the combined gain is a combined band gain associated with a plurality of bands of the audio signal, the method further comprising:

[0115] converting the combined band gain to a combined bin gain, wherein the combined bin gain is associated with a plurality of bins.

[0116] EEE 14. A non-transitory computer-readable medium storing a computer program that, when executed by a processor, controls an apparatus to perform processing comprising the method of any of EEEs 1-13.

[0117] EEE 15. An apparatus for audio processing, the apparatus comprising:

[0118] a processor; and

[0119] a memory,

[0120] wherein the processor is configured to control the apparatus to generate, using a machine learning model, first band gains and a voice activity detection value for an audio signal;

[0121] wherein the processor is configured to control the apparatus to generate, based on the first band gains and the voice activity detection value, a background noise estimate;

[0122] wherein the processor is configured to control the apparatus to generate the second band gain by processing the audio signal using a Wiener filter controlled by the background noise estimate;

[0123] wherein the processor is configured to control the apparatus to generate the combined gain by combining the first band gain and the second band gain; and

[0124] wherein the processor is configured to control the apparatus to generate the modified audio signal by modifying the audio signal using the combined gain.

[0125] EEE 16. The apparatus of EEE 15, wherein the machine learning model is generated using data augmentation to increase diversity of training data.

[0126] EEE 17. The apparatus of any of EEEs 15-16, wherein at least one restriction is applied in generating at least one of the first band gain and the second band gain.

[0127] EEE 18. The apparatus of any of EEEs 15-17, wherein generating the background noise estimate is based on a number of noise frames exceeding a threshold for a particular band.

[0128] EEE 19. The apparatus of any of EEEs 15-18, wherein the processor is configured to control the apparatus to perform a spectral analysis on the audio signal to generate a plurality of bin features and a fundamental frequency of the audio signal, and

[0129] wherein the first band gain and the voice activity detection value are based on the plurality of bin features and the fundamental frequency.

[0130] EEE 20. The apparatus of EEE 19, wherein the processor is configured to control the apparatus to generate a plurality of band features based on the plurality of bin features, wherein the plurality of band features are generated using one of a mel frequency cepstral coefficient and a bark frequency cepstral coefficient, and

[0131] wherein the first band gain and the voice activity detection value are based on the plurality of band features and the fundamental frequency.

[0132] REFERENCES

[0133] U.S. Patent Application Publication No. 2019 / 0378531.

[0134] U.S. Patent Nos. 10,546,593 B2; 10,224,053 B2; 9,053,697 B2.

[0135] Chinese Patent Publication Nos. CN105513605B; CN111192599A; CN110660407B; CN110211598A; CN110085249A; CN109378013A; CN109065067A; CN107863099A.

[0136] Jean-Marc Valin, “A Hybrid DSP Deep Learning Approach to Real-Time Full-Band Speech Enhancement,” 2018 IEEE 20th International Symposium on Multimedia Signal Processing (MMSP), DOI: 10.1109 / MMSP.2018.8547084.

[0137] Xia, Y., Stern, R., “A Priori SNR Estimation Based on a Recurrent Neural Network for Robust Speech Enhancement,” Proc. Interspeech 2018, 3274-3278, DOI: 10.21437 / Interspeech.2018-2423.

[0138] Zhang, Q., Nicolson, A. M., Wang, M., Paliwal, K., & Wang, C.-X., “Deep MMSE: A Deep Learning Approach to MMSE-based Noise Power Spectral Density Estimation,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, 1-1. DOI: 10.1109 / taslp.2020.2987441.

Claims

1. A computer-implemented method of audio processing, the method comprising: generating, using a machine learning model, first band gains and voice activity detection values for an audio signal; generating, based on the first band gains and the voice activity detection values, a background noise estimate; generating second band gains by processing the audio signal using a Wiener filter controlled by the background noise estimate; generating combined gains by combining the first band gains and the second band gains, including: multiplying the first band gains and the second band gains; and limiting the combined gains using at least two different limits for at least two different bands; and generating a modified audio signal by modifying the audio signal using the combined gains.

2. The method of claim 1, wherein, The machine learning model is generated using data augmentation to increase diversity of training data.

3. The method of any one of claims 1-2, wherein, Generating the first band gains includes limiting the first band gains using at least two different limits for at least two different bands.

4. The method of any one of claims 1-2, wherein, Generating the background noise estimate is based on a number of noise frames exceeding a threshold for a particular band.

5. The method of any one of claims 1-2, wherein, Generating the second band gains includes using the Wiener filter based on a steady state noise level for a particular band.

6. The method of any one of claims 1-2, wherein, Generating the second band gains includes limiting the second band gains using at least two different limits for at least two different bands.

7. The method of any one of claims 1-2, wherein, Generating the modified audio signal includes modifying an amplitude spectrum of the audio signal using the combined gains.

8. The method of any of claims 1-2, further comprising: applying an overlap window to an input audio signal to generate a plurality of frames, wherein the audio signal corresponds to the plurality of frames.

9. The method of any of claims 1-2, further comprising: performing a spectral analysis on the audio signal to generate a plurality of bin features and a fundamental frequency for the audio signal, wherein the first band gains and the voice activity detection values are based on the plurality of bin features and the fundamental frequency.

10. The method of claim 9, further comprising: generating a plurality of band features based on the plurality of bin features, wherein the plurality of band features are generated using one of a mel frequency cepstral coefficient and a bark frequency cepstral coefficient, wherein the first band gains and the voice activity detection values are based on the plurality of band features and the fundamental frequency.

11. The method of any one of claims 1-2, wherein, The combined gains are combined band gains associated with a plurality of bands of the audio signal, the method further comprising: converting the combined band gains to combined bin gains, wherein the combined bin gains are associated with a plurality of bins.

12. A non-transitory computer-readable medium storing a computer program that, when executed by a processor, controls a device to perform processing including the method of any of claims 1-11.

13. A device for audio processing, the device comprising: a processor; and a memory, wherein the processor is configured to control the device to generate, using a machine learning model, first band gains and voice activity detection values for an audio signal; ​ wherein the processor is configured to control the apparatus to generate a background noise estimate based on the first band gain and the voice activity detection value; wherein the processor is configured to control the apparatus to generate a second band gain by processing the audio signal using a Wiener filter controlled by the background noise estimate; wherein the processor is configured to control the apparatus to generate a combined gain by combining the first band gain and the second band gain, wherein generating the combined gain comprises: multiplying the first band gain and the second band gain; and limiting the combined gain using at least two different limits for at least two different frequency bands; and wherein the processor is configured to control the apparatus to generate a modified audio signal by modifying the audio signal using the combined gain.

14. The apparatus of claim 13, wherein, applying at least one limit when generating at least one of the first band gain and the second band gain.

15. A computer program product comprising a computer program which comprises instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1-11.

Images