Noise reduction using machine learning

Abstract

SOLUTION: A noise reduction system 100 includes controlling a Wiener filter 110 using a neural network 108. Gain 158 estimated by the neural network is combined with gain 162 generated by the Wiener filter by a gain combination block 112 to generate gain 164.EFFECT: A noise reduction system 100 provides improved results compared to using a neural network alone.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to European Patent Application No. 20206921.7, filed November 11, 2020, U.S. Provisional Patent Application No. 63 / 110,114, filed November 5, 2020, U.S. Provisional Patent Application No. 63 / 068,227, filed August 20, 2020, and International Patent Application No. PCT / CN2020 / 106270, filed July 31, 2020, all of which are incorporated herein by reference in their entirety.

[0002] Field The present disclosure relates to audio processing, and in particular to noise reduction. [Background technology]

[0003] Unless otherwise stated herein, the approaches described in this section are not prior art to the claims of this application and are not admitted to be prior art by their inclusion in this section.

[0004] Noise reduction is difficult to implement on mobile devices. Mobile devices can capture both stationary and non-stationary noise in a variety of use cases, including voice communications, user-generated content development, etc. Because mobile devices can be constrained in power consumption and processing capabilities, it is difficult to develop a noise reduction process that is effective when implemented by a mobile device. Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above, there is a need to develop a noise reduction system that works well in mobile devices. [Means for solving the problem]

[0006] According to one embodiment, a computer-implemented audio processing method includes generating a first-band gain and a voice activity detection value for an audio signal using a machine learning model. The method further includes generating a background noise estimate based on the first-band gain and the voice activity detection value. The method further includes generating a second-band gain by processing the audio signal using a Wiener filter controlled by the background noise estimate. The method further includes generating a combined gain by combining the first-band gain and the second-band gain. The method further includes generating a modified audio signal by modifying the audio signal using the combined gain.

[0007] According to another embodiment, an apparatus includes a processor and a memory, the processor configured to control the apparatus to implement one or more of the methods described herein, and the apparatus may further include similar details to one or more of the methods described herein.

[0008] According to another embodiment, a non-transitory computer-readable medium stores a computer program that, when executed by a processor, controls an apparatus to perform processes including one or more of the methods described herein.

[0009] The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of a noise reduction system 100.

[0011] [Figure 2] 1 is a block diagram of an example of a system 200 suitable for implementing exemplary embodiments of the present disclosure.

[0012] [Figure 3]3 is a flow diagram of a method 300 of audio processing. DETAILED DESCRIPTION OF THE INVENTION

[0013] This application describes techniques related to noise reduction. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure, as defined by the claims, may include some or all of the features of these examples, alone or in combination with other features described below, and may also include modifications and equivalents of the features and concepts described herein.

[0014] The following description details various methods, processes, and procedures. While specific steps may be described in a certain order, such order is primarily for convenience. Certain steps may be repeated multiple times, may occur before or after other steps (even if those steps are described in a different order), or may occur in parallel with other steps. A second step is required to follow a first step only if the first step must be completed before the second step can begin. Such situations will be specifically pointed out if not clear from the context.

[0015] The terms "and," "or," and "and / or" are used herein. Such terms should be read as inclusive. For example, "A and B" can mean at least the following: "both A and B," "at least both A and B." As another example, "A or B" can mean at least the following: "at least A," "at least B," "both A and B," "at least both A and B." As another example, "A and / or B" can mean at least the following: "A and B," "A or B." If an exclusive disjunction is intended, this will be specifically stated (e.g., "either A or B," "at most one of A and B").

[0016] This document describes various processing functions associated with structures such as blocks, elements, components, circuits, etc. Generally, these structures may be implemented by a processor controlled by one or more computer programs.

[0017] FIG. 1 is a block diagram of a noise reduction system 100. The noise reduction system 100 may be implemented in a mobile device (e.g., see FIG. 2 ), such as a mobile phone or a video camera with a microphone. The components of the noise reduction system 100 may be implemented, for example, by a processor controlled according to one or more computer programs. The noise reduction system 100 includes a windowing block 102, a transform block 104, a band feature analysis block 106, a neural network 108, a Wiener filter 110, a gain combination block 112, a band gain vs. bin gain block 114, a signal modification block 116, an inverse transform block 118, and an inverse windowing block 120. The noise reduction system 100 may include other components not described in detail (for brevity).

[0018] The windowing block 102 receives an audio signal 150 and performs windowing on the audio signal 150 to generate audio frames 152. The audio signal 150 may be captured by a microphone of a mobile device implementing the noise reduction system 100. Generally, the audio signal 150 is a time-domain signal including a sequence of audio samples. For example, the audio signal 150 may be captured at a sampling rate of 48 kHz, with each sample quantized at a bit rate of 16 bits. Other exemplary sampling rates may include 44.1 kHz, 96 kHz, 192 kHz, etc., and other bit rates may include 24 bits, 32 bits, etc.

[0019] In general, the windowing block 102 applies overlapping windows to samples of the audio signal 150 to generate audio frames 152. The windowing block 102 can implement various forms of windowing, including rectangular windows, triangular windows, trapezoidal windows, sine windows, etc.

[0020] The transform block 104 receives audio frames 152 and performs a transform on the audio frames 152 to generate transform features 154. The transform may be a frequency-domain transform, and the transform features 154 may include bin features and fundamental frequency parameters for each audio frame. (The transform features 154 are sometimes referred to as bin features 154.) The fundamental frequency parameters may include the fundamental frequency of the audio, referred to as F0. The transform block 104 may implement various transforms, including Fourier transforms (e.g., fast Fourier transforms (FFTs)), quadrature mirror filter (QMF) domain transforms, and the like. For example, the transform block 104 may implement an FFT with a 960-point decomposition window and a 480-point frame shift; alternatively, a 1024-point decomposition window and a 512-point frame shift may be implemented. The number of bins in the transform features 154 is generally related to the number of points in the transform decomposition. For example, a 960-point FFT results in 481 bins.

[0021] The transform block 104 can implement various processes to determine the fundamental frequency parameters of each audio frame. For example, if the transform is an FFT, the transform block 104 can extract the fundamental frequency parameters from the FFT parameters. As another example, the transform block 104 can extract the fundamental frequency parameters based on autocorrelation of the time-domain signal (e.g., audio frame 152).

[0022] The band feature analysis block 106 receives the transformed features 154 and performs band analysis on the transformed features 154 to generate band features 156. The band features 156 may be generated according to various scales, including the Mel scale, the Bark scale, etc. The number of bands in the band features 156 may vary when using different scales, for example, 24 bands for the Bark scale, 80 bands for the Mel scale, etc. The band feature analysis block 106 may combine the band features 156 with a fundamental frequency parameter (e.g., F0).

[0023] The band feature analysis block 106 can use rectangular bands. The band feature analysis block 106 can also use triangular bands with peak responses at the boundaries between bands.

[0024] The band features 156 may be band energies such as Mel band energy, Bark band energy, etc. The band feature analysis block 106 may calculate logarithmic values of the Mel band energy and the Bark band energy. The band feature analysis block 106 may apply a discrete cosine transform (DCT) of the band energies to generate new band features, such that the new band features are less correlated than the original band features. For example, the band feature analysis block 106 may generate the band features 156 as Mel-frequency cepstral coefficients (MFCCs), Bark-frequency cepstral coefficients (BFCCs), etc.

[0025] The band feature analysis block 106 may perform smoothing of the current frame and previous frames according to a smoothing value. The band feature analysis block 106 can also perform differential analysis by calculating first and second order differentials between the current frame and previous frames.

[0026] The band feature analysis block 106 may calculate a band harmonicity feature that indicates how much of the current band is composed of periodic signals. For example, the band feature analysis block 106 may calculate the band harmonicity feature based on the FFT frequency bind of the current frame. As another example, the band feature analysis block 106 may calculate the band harmonicity feature based on the correlation between the current frame and the immediately preceding frame.

[0027] Generally, the band features 156 are fewer in number than the bin features 154, thus reducing the dimensionality of the data input to the neural network 108. For example, the bin features may be on the order of 513 or 481 bins, and the band features 156 may be on the order of 24 or 80 bands.

[0028] The neural network 108 receives the band features 156, processes the band features 156 according to the model, and generates a gain 158 and a voice activity decision (VAD) 160. The gain 158 is sometimes referred to as DGain, for example, to indicate that it is the output of the neural network. The model is trained offline. Training the model, including preparing a training data set, is described in a later section.

[0029] The neural network 108 uses this model to estimate gain and voice activity for each band based on band features 156 (e.g., including fundamental frequency F0) and outputs gain 158 and VAD 160. The neural network 108 may be a fully connected neural network (FCNN), a recurrent neural network (RNN), a convolutional neural network (CNN), another type of machine learning system, etc., or a combination thereof.

[0030] The noise reduction system 100 may apply smoothing or limiting to the DGains output of the neural network 108. For example, the noise reduction system 100 may apply average smoothing or median filtering to the gains 158 along a time axis, a frequency axis, etc. As another example, the noise reduction system 100 may apply limiting to the gains 158, with a maximum gain of 1.0 and minimum gains that are different for different bands. In one implementation, the noise reduction system 100 sets a minimum gain of 0.1 (e.g., −20 dB) for the lowest four bands and a minimum gain of 0.18 (e.g., −15 dB) for the middle band. Setting the minimum gains reduces discontinuities in the DGains. The minimum gain values may be adjusted as desired. For example, minimum gains of −12 dB, −15 dB, −18 dB, −20 dB, etc. may be set for various bands.

[0031] The Wiener filter 110 receives the band features 156, the gain 158, and the VAD 160, performs Wiener filtering, and generates a gain 162. The gain 162 may be referred to as WGains, for example, to indicate that it is the output of the Wiener filter. Generally, the Wiener filter 110 estimates the background noise in each band of the input signal 150 according to the band features 156. (Background noise is sometimes referred to as stationary noise.) The Wiener filter 110 controls its filtering process using the gain 158 and the VAD 160 estimated by the neural network. In one implementation, for a given input frame (with a corresponding band feature 156) without voice activity (e.g., VAD 160 less than 0.5), the Wiener filter 110 checks the band gain for the given input frame (according to the gain 158 (DGains)). For bands with DGains less than 0.5, the Wiener filter 110 considers these bands as noise frames and smooths the band energy of these frames to obtain an estimate of the background noise.

[0032] The Wiener filter 110 may track the average number of frames used to calculate the band energy for each band to obtain a noise estimate. If the average number for a given band is greater than a threshold number of frames, the Wiener filter 110 is applied to calculate a Wiener band gain for the given band. If the average number for a given band is less than the threshold number of frames, the Wiener band gain is 1.0 for the given band. The Wiener band gain for each band is output as gain 162, also known as the Wiener gain (or WGains).

[0033] In effect, the Wiener filter 110 estimates the background noise in each band based on the signal history (e.g., several frames of the input signal 150). The frame count threshold provides the Wiener filter 110 with a sufficient number of frames that will lead to a reliable estimate of the background noise. In one implementation, the frame count threshold is 50. If a frame is 10 ms, this corresponds to 0.5 seconds of the input signal 150. If the frame count is less than the threshold (e.g., WGains is 1.0), the Wiener filter 110 is effectively bypassed.

[0034] The noise reduction system 100 may apply limiting to the WGains output of the Wiener filter 110, with a maximum gain of 1.0 and a minimum gain that varies for different bands. In one implementation, the noise reduction system 100 sets a minimum gain of 0.1 (e.g., −20 dB) for the lowest four bands and a minimum gain of 0.18 (e.g., −15 dB) for the mid-band. Setting the minimum gain mitigates discontinuities in the WGains. The minimum gain value may be adjusted as desired. For example, minimum gains of −12 dB, −15 dB, −18 dB, −20 dB, etc. may be set for various bands.

[0035] Gain combiner block 112 receives gains 158 (DGains) and gains 162 (WGains) and combines them to generate gains 164. Gain 164 may also be referred to as band gain, combined band gain, or CGains, for example, to indicate that it is a combination of DGains and WGains. As an example, gain combiner block 112 may multiply DGains and WGains to generate CGains for each band.

[0036] The noise reduction system 100 may apply limiting to the CGains output of the gain combination block 112, with a maximum gain of 1.0 and a minimum gain that varies for different bands. In one implementation, the noise reduction system 100 sets a gain of 0.1 (e.g., −20 dB) as the minimum gain for the lowest four bands and a gain of 0.18 (e.g., −15 dB) as the minimum gain for the mid-band. Setting the minimum gain mitigates discontinuities in the CGains. The minimum gain value may be adjusted as desired. For example, minimum gains of −12 dB, −15 dB, −18 dB, −20 dB, etc. may be set for various bands.

[0037] Band gain to bin gain block 114 receives gain 164 and converts the band gain to a bin gain to generate gain 166 (also referred to as the bin gain). In effect, band gain to bin gain block 114 performs the inverse of the process performed by band feature analysis block 106 to convert gain 164 from a band gain to a bin gain. For example, if band feature analysis block 106 processed 1024-point FFT bins into 24 Bark-scale bands, band gain to bin gain block 114 converts the 24 Bark-scale bands of gain 164 into 1024 FFT bins of gain 166.

[0038] The band gain to bin gain block 114 can implement various techniques for converting the band gain to the bin gain. For example, the band gain to bin gain block 114 can use interpolation, such as linear interpolation.

[0039] The signal modification block 116 receives the transform features 154 (including the bin features and the fundamental frequency F) and the gain 166, modifies the transform features 154 according to the gain 166, and generates modified transform features 168 (including the modified bin features and the fundamental frequency F). (The modified transform features 168 are sometimes referred to as modified bin features 168.) The signal modification block 116 may modify the amplitude spectrum of the bin features 154 based on the gain 166. In one implementation, the signal modification block 116 leaves the phase spectrum of the bin features 154 unchanged when generating the modified bin features 168. In another implementation, the signal modification block 116 adjusts the phase spectrum of the bin features 154 when generating the modified bin features 168, for example, by performing an estimation based on the modified bin features 168. As an example, the signal modification block 116 can use a short-time Fourier transform to adjust the phase spectrum, for example, by implementing a Griffin-Lim process.

[0040] The inverse transform block 118 receives the modified transform features 168 and performs an inverse transform on the modified transform features 168 to generate audio frames 170. Generally, the inverse transform performed is the inverse of the transform performed by the transform block 104. For example, the inverse transform block 118 may implement an inverse Fourier transform (e.g., an inverse FFT), an inverse QMF transform, etc.

[0041] Inverse windowing block 120 receives audio frame 170 and performs inverse windowing on audio frame 170 to generate audio signal 172. Generally, the inverse windowing performed is the inverse of the windowing performed by windowing block 102. For example, inverse windowing block 120 may perform overlap and add on audio frame 170 to generate audio signal 172.

[0042] As a result, the combination of using the output of the neural network 108 to control the Wiener filter 110 may provide improved results over simply using a neural network alone to perform noise reduction, as many neural networks simply operate with a short memory.

[0043] 2 illustrates a block diagram of an exemplary system 200 suitable for implementing exemplary embodiments of the present disclosure. System 200 includes one or more server computers or any client device. System 200 may include any consumer device, including, but not limited to, a smartphone, a media player, a tablet computer, a laptop, a wearable computer, a vehicle computer, a game console, a surround system, a kiosk, etc.

[0044] As shown, system 200 includes a central processing unit (CPU) 201 that can perform various processes according to programs stored, for example, in read-only memory (ROM) 202 or loaded, for example, from storage unit 208 into random access memory (RAM) 203. RAM 203 also stores data needed by CPU 201 to perform various processes, as needed. CPU 201, ROM 202, and RAM 203 are connected to one another via bus 204. Input / output (I / O) interface 205 is also connected to bus 204.

[0045] The following components are connected to I / O interface 205: input unit 206, which may include a keyboard, mouse, touch screen, motion sensor, camera, etc.; output unit 207, which may include a display, such as a liquid crystal display (LCD), and one or more speakers; storage unit 208, which may include a hard disk or other suitable storage device; and communication unit 209, which may include a network interface card, such as a network card (e.g., wired or wireless). Communication unit 209 may also communicate with wireless input / output components, such as, for example, a wireless microphone, wireless earphones, wireless speakers, etc.

[0046] In some implementations, the input unit 206 includes one or more microphones in different locations (depending on the host device) that enable the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, or other suitable formats).

[0047] In some implementations, output unit 207 includes a system with a varying number of speakers. As shown in Figure 2, output unit 207 can render audio signals in a variety of formats (e.g., mono, stereo, immersive, binaural, or other suitable formats) (depending on the capabilities of the host device).

[0048] The communication unit 209 is configured to communicate with other devices (e.g., via a network). If necessary, a drive 210 is also connected to the I / O interface 205. A removable medium 211, such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or other suitable removable medium, is mounted on the drive 210, and a computer program read from the removable medium 211 is installed in the storage unit 208, if necessary. While the system 200 has been described as including the above components, those skilled in the art will understand that, in actual applications, some of these components may be added, removed, and / or substituted, and all such modifications or variations are within the scope of the present disclosure.

[0049] For example, system 200 may implement one or more components of noise reduction system 100 (see FIG. 1 ), e.g., by executing one or more computer programs on CPU 201. ROM 802, RAM 803, storage unit 808, etc. may store models used by neural network 108. A microphone connected to input device 206 may capture audio signal 150, and a speaker connected to output device 207 may output sound corresponding to audio signal 172.

[0050] 3 is a flow diagram of an audio processing method 300. The method 300 may be implemented by an apparatus (e.g., system 200 of FIG. 2) as controlled by the execution of one or more computer programs.

[0051] At 302, a first band gain and a voice activity detection value for the audio signal are generated using a machine learning model. For example, CPU 201 may implement neural network 108 (see FIG. 1) to generate gain 158 and VAD 160 by processing band features 156 according to the model.

[0052] At 304, a background noise estimate is generated based on the first band gain and the voice activity detection value. For example, the CPU 201 may generate the background noise estimate based on the gain 158 and the VAD 160 as part of operating the Wiener filter 110.

[0053] At 306, a second band gain is generated by processing the audio signal with a Wiener filter controlled by the background noise estimate. For example, CPU 201 may implement Wiener filter 110 to generate gain 162 by processing band features 156 controlled by the background noise estimate (see 304). For example, when the number of noise frames exceeds a threshold (e.g., 50 noise frames) for a particular band, the Wiener filter generates a second band gain for that particular band.

[0054] At 308, a combined gain is generated by combining the first band gain and the second band gain. For example, CPU 201 may implement gain combination block 112 to generate gain 164 by combining gain 158 (from neural network 108) and gain 162 (from Wiener filter 110). The first band gain and the second band gain may be combined by multiplication. The first band gain and the second band gain may be combined by selecting the maximum value of the first band gain and the second band gain for each band. Limiting may be applied to the combined gain. The first band gain and the second band gain may be combined by multiplication or by selecting the maximum value for each band, and limiting may be applied to the combined gain.

[0055] At 310, a modified audio signal is generated by modifying the audio signal using the combined gains. For example, the CPU 201 can implement the signal modification block 116 to generate modified bin features 168 by modifying the bin features 154 using the gains 166.

[0056] The method 300 may include other steps similar to those described above with respect to the noise reduction system 100. A non-exhaustive discussion of exemplary steps includes the following: A windowing step (see windowing block 102) may be performed on the audio signal as part of generating an input to the neural network 108. A transform step (see transform block 104) may be performed on the audio signal to convert time-domain information to frequency-domain information as part of generating an input to the neural network 108. A bin-to-band transform step (see band feature analysis block 106) may be performed on the audio signal to reduce the dimensionality of the input to the neural network 108. A band-to-bin transform step (see band gain to bin gain block 114) may be performed to convert band gains (e.g., gain 164) to bin gains (e.g., gain 166). An inverse transform step (see inverse transform block 118) may be performed to convert the modified bin features 168 from frequency-domain information to time-domain information (e.g., audio frames 170). An inverse windowing step (see inverse windowing block 120) may be performed to reconstruct the audio signal 172 as the inverse of the windowing step.

[0057] Creating a Model

[0058] As previously mentioned, the model used in the neural network 108 (see FIG. 1) may be trained offline and then stored and used by the noise reduction system 100. For example, a computer system may implement a model training system that trains the model, e.g., by executing one or more computer programs. Part of training the model includes preparing training data to generate input features and target features. The input features may be calculated by band feature calculation of the noisy data (X). The target features consist of ideal band gains and VAD decisions.

[0059] The noisy data (X) can be generated by combining clean speech (S) and noisy data (N).

[0060] X=S+N The VAD decision may be based on an analysis of the clean speech S. In some implementations, the VAD decision is determined by an absolute threshold of the energy of the current frame. In other implementations, other VAD methods may be used. For example, the VAD may be manually labeled.

[0061] The ideal band gain g is calculated by the following formula:

[0062] g b =√(E s (b) / E x (b)) where Es(b) is the energy in band b of clean speech, and E x (b) Energy in band b of the noisy speech.

[0063] To make the model robust to different use cases, the model training system may perform data augmentation on the training data. i and N i Given an input utterance file with S, the model training system will first mix S before mixing with the noisy data. i and N i Data augmentation involves three general steps:

[0064] The first step is to control the amplitude of clean speech. A common problem for noise reduction models is suppressing low-volume speech. Thus, model training systems perform data augmentation by preparing training data containing speech of various amplitudes.

[0065] The model training system sets a random target mean amplitude in the range of -45 dB to 0 dB (e.g., -45, -40, -35, -30, -25, -20, -15, -10, -5, 0). The model training system modifies the input utterance file by the value a to match the target mean amplitude. S m =a*S i

[0066] The second step is to control the signal-to-noise ratio (SNR). For each combination of speech and noise files, the model training system sets a random target SNR. In one implementation, the target SNR is randomly selected with equal probability from the set of SNRs [-5, -3, 0, 3, 5, 10, 15, 18, 20, 30]. The model training system then modifies the input noise file by a value b to obtain S m N m Match the SNR between N m =b*N i

[0067] The third step is to constrain the mixed data. The model training system first constrains the mixed signal X by the following equation: m Calculate. X m =(S m +N m )

[0068] If clipping occurs (e.g., X as a .wav file with 16-bit quantization) m ), the model training system uses A max Marked with X m Calculate the maximum absolute value of

[0069] The correction ratio c can then be calculated by: c=32767 / A max

[0070] In the above formula, the value 32767 comes from 16-bit quantization; this value can be adjusted as needed for other bit quantization precisions.

[0071] Next, S=c*S m N=c*N m

[0072] S and N are mixed into noisy speech X. X=S+N

[0073] The calculation of the average amplitude and SNR may be performed according to various processes as desired: The model training system may use a minimum threshold to remove silent segments before calculating the average amplitude.

[0074] Thus, data augmentation is used to increase the diversity of the training data by conditioning segments of the training data using a variety of target mean amplitudes and target SNRs. For example, using 10 variations of the target mean amplitude and 10 variations of the target SNR results in 100 variations of a single segment of training data. Data augmentation does not necessarily increase the size of the training data. If the training data is 100 hours before data augmentation, the full set of 10,000 hours of augmented training data does not need to be used to train the model; the augmented training data set may be limited to a smaller size, e.g., 100 hours. More importantly, data augmentation increases the variability of amplitude and SNR in the training data.

[0075] Implementation details

[0076] Embodiments may be implemented in hardware, in executable modules stored on a computer-readable medium, or in a combination of both (e.g., a programmable logic array). Unless otherwise specified, steps performed by embodiments need not inherently relate to any particular computer or other apparatus, although in certain embodiments they may. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments may be implemented in one or more computer programs running on one or more programmable computer systems, each including at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The program code is applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in known fashion.

[0077] Each such computer program is preferably stored on or downloaded to a general-purpose or special-purpose programmable computer-readable storage medium or device (e.g., solid-state memory or media, magnetic or optical media) so as to configure and operate a computer to perform the procedures described herein when the storage medium or device is read by a computer system. The system of the present invention is also considered to be implemented as a computer-readable storage medium configured with a computer program. The storage medium so configured causes a computer system to operate in a specific, predefined manner to perform the functions described herein. (Software itself, and intangible or transitory signals, are excluded, insofar as they are non-patentable subject matter.)

[0078] The above description illustrates various embodiments of the present disclosure, along with examples of how aspects of the disclosure may be implemented. The above examples and embodiments should not be considered the only embodiments, but are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations, and equivalents will be apparent to those skilled in the art and may be adopted without departing from the spirit and scope of the present disclosure as defined by the claims.

[0079] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE). [EEE1] 1. A computer-implemented audio processing method, the method comprising: Using the machine learning model to generate a first band gain and a voice activity detection value for the audio signal; generating a background noise estimate based on the first band gain and the voice activity detection; generating a second band gain by processing the audio signal with a Wiener filter controlled by the background noise estimate; generating a combined gain by combining the first band gain and the second band gain; modifying the audio signal using the combined gains to generate a modified audio signal. method. [EEE2] The method of EEE1, wherein the machine learning model is generated using data augmentation to increase diversity of training data. [EEE3] The method of EEE1 or EEE2, wherein generating the first band gain and the voice activity detection value is performed using one of a fully connected neural network, a recurrent neural network, and a convolutional neural network. [EEE4] 4. The method of any one of claims 1 to 3, wherein generating the first band gain comprises limiting the first band gain using at least two different limits for at least two different bands. [EEE5] 5. The method of any one of EEE1 to 4, wherein generating the background noise estimate is based on a number of noise frames that exceed a threshold for a particular band. [EEE6] 6. The method of any one of claims 1 to 5, wherein generating the second band gain comprises using the Wiener filter based on a steady-state noise level for a particular band. [EEE7] 7. The method of any one of EEE1 to 6, wherein generating the second-band gain comprises limiting the second-band gain using at least two different limits for at least two different bands. [EEE8] Generating the combined gain includes: multiplying the first band gain and the second band gain; limiting the combined band gain using at least two different limits for at least two different bands. The method according to any one of EEE1 to 7. [EEE9] The method of any one of EEE1 to 8, wherein generating the modified audio signal comprises modifying an amplitude spectrum of the audio signal using the combined band gains. [EEE10] 10. The method of any one of EEE1 to 9, further comprising applying an overlapping window to an input audio signal to generate a plurality of frames, the audio signal corresponding to the plurality of frames. [EEE11] performing a spectral analysis on the audio signal to generate a plurality of bin features and a fundamental frequency of the audio signal; the first band gain and the voice activity detection value are based on the plurality of bin features and the fundamental frequency. The method of any one of EEE1 to EEE10. [EEE12] generating a plurality of band features based on the plurality of bin features, the plurality of band features being generated using one of Mel-frequency cepstral coefficients and Bark-frequency cepstral coefficients; the first band gain and the voice activity detection value are based on the plurality of band features and the fundamental frequency. Method according to EEE11. [EEE13] The combined gain is a combined band gain associated with a plurality of bands of the audio signal, the method further comprising: converting the combined band gains to combined bin gains, the combined bin gains being associated with a plurality of bins; The method of any one of EEE1 to EEE12. [EEE14] A non-transitory computer readable medium storing a computer program which, when executed by a processor, controls an apparatus to perform processes including the methods described in any one of EEE1 to EEE13. [EEE15] 1. An apparatus for audio processing, comprising: processor; and It has a memory, the processor is configured to control the device to generate a first band gain and a voice activity detection value for the audio signal using a machine learning model; the processor is configured to control the apparatus to generate a background noise estimate based on the first band gain and the voice activity detection; the processor is configured to control the apparatus to generate a second band gain by processing the audio signal using a Wiener filter controlled by the background noise estimate; the processor is configured to control the apparatus to generate a combined gain by combining the first band gain and the second band gain; the processor is configured to control the device to generate a modified audio signal by modifying the audio signal using the combined gain. Device. [EEE16] The apparatus of EEE16, wherein the machine learning model is generated using data augmentation to increase diversity of training data. [EEE17] 17. The apparatus of claim 15 or 16, wherein at least one limitation is applied when generating at least one of the first band gain and the second band gain. [EEE18] 18. The apparatus of any one of EEE15 to 17, wherein generating the background noise estimate is based on a number of noise frames that exceed a threshold for a particular band. [EEE19] the processor is configured to control the apparatus to perform a spectral analysis on the audio signal to generate a plurality of bin features and a fundamental frequency of the audio signal; the first band gain and the voice activity detection value are based on the plurality of bin features and the fundamental frequency. 8. Apparatus according to any one of claims EEE15 to 18. [EEE20] the processor is configured to control the apparatus to generate a plurality of band features based on the plurality of bin features, the plurality of band features being generated using one of Mel-frequency cepstral coefficients and Bark-frequency cepstral coefficients; the first band gain and the voice activity detection value are based on the plurality of band features and the fundamental frequency. The device described in EEE19. [Prior art documents] [Patent documents]

[0080] [Patent Document 1] U.S. Patent Application Publication No. 2019 / 0378531 [Patent Document 2] U.S. Patent No. 10,546,593B2 [Patent Document 3] U.S. Patent No. 10,224,053B2 [Patent Document 4] U.S. Patent No. 9,053,697B2 [Patent Document 5] China Patent Publication No. 105513605B [Patent Document 6] China Patent Publication No. 111192599A [Patent Document 7] China Patent Publication No. 110660407B [Patent Document 8] China Patent Publication No. 110211598A [Patent Document 9] China Patent Publication No. 110085249A [Patent Document 10] China Patent Publication No. 109378013A [Patent Document 11] China Patent Publication No. 109065067A [Patent Document 12] China Patent Publication No. 107863099A [Non-patent literature]

[0081] [Non-Patent Document 1] Jean-Marc Valin, "A Hybrid DSP Deep Learning Approach to Real-Time Full-Band Speech Enhancement", 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), DOI: 10.1109 / MMSP.2018.8547084. [Non-Patent Document 2] Xia, Y., Stern, R., "A Priori SNR Estimation Based on a Recurrent Neural Network for Robust Speech Enhancement", Proc. Interspeech 2018, 3274-3278, DOI: 10.21437 / Interspeech.2018-2423. [Non-Patent Document 3] Zhang, Q., Nicolson, A. M., Wang, M., Paliwal, K., & Wang, C.-X., "DeepMMSE: A Deep Learning Approach to MMSE-based Noise Power Spectral Density Estimation", IEEE / ACM Transactions on Audio, Speech, and Language Processing, 1-1. DOI:10.1109 / taslp.2020.2987441.

Claims

1. 1. A computer-implemented method of audio processing, the method comprising: determining a first characteristic of the audio signal; generating a second feature of the audio signal via a neural network model; processing the audio signal based on the first feature and the second feature to determine a modified audio signal; A computer-implemented method.

2. 10. The computer-implemented method of claim 1, wherein the neural network model comprises one of a recurrent neural network, a convolutional neural network, and / or a fully connected neural network.

3. The computer-implemented method of claim 1 or 2, wherein the second features include at least a voice activity value and / or a gain value.

4. The computer-implemented method of claim 1 , wherein the first feature comprises a band feature.

5. The computer-implemented method of claim 4 , wherein the band features include band energy.

6. 6. The computer-implemented method of claim 1, wherein the processing is performed at least in part by a Wiener filter.

7. Determining the first feature of the audio signal comprises: performing a transform on frames of the audio signal to generate transform features; performing a band analysis on the transformed features to determine the first features; A computer-implemented method according to any one of claims 1 to 6.

8. The computer-implemented method of claim 7 , wherein the transform features include at least one of bin features or fundamental frequency parameters.

9. 9. The computer-implemented method of claim 1, wherein the processing further comprises determining a third feature of the audio signal based on the first and second features.

10. The computer-implemented method of claim 9 , wherein the third feature comprises a gain value.

11. 11. The computer-implemented method of claim 9 or 10, wherein processing the audio signal based on the first feature and the second feature to determine a modified audio signal comprises modifying a transformation feature of the audio signal based on the third feature to determine the modified audio signal.

12. A non-transitory computer readable medium storing a computer program which, when executed by a processor, controls an apparatus to perform processes including the method of any one of claims 1 to 11.

13. 1. An apparatus for audio processing, comprising: A processor including a neural network model, the processor comprising: determining a first characteristic of the audio signal; generating a second feature of the audio signal via the neural network model; Processing the audio signal based on the first feature and the second feature to determine a modified audio signal. The apparatus is configured to:

Images