An audio noise reduction model training method, device and storage medium

Abstract

The application discloses an audio noise reduction model training method and device and a storage medium, and relates to the technical field of audio processing. The method comprises the following steps: acquiring noisy audio, inputting the noisy audio into a target neural network for training with clean audio as a training target; the target neural network takes the noisy audio as input and takes noise reduction audio as output; a signal-to-noise ratio estimator is used to determine the signal-to-noise ratio of the noise reduction audio corresponding to the noisy audio as a first signal-to-noise ratio; a first loss function is determined according to the first signal-to-noise ratio and a second signal-to-noise ratio corresponding to the noisy audio; the target neural network is iteratively trained by using the first loss function and the second loss function, and a trained audio noise reduction model is obtained; the second loss function is constructed according to the difference between the clean audio corresponding to the noisy audio and the noise reduction audio. The audio difference and the signal-to-noise ratio difference are combined for joint constraint, the audio noise reduction capability is improved, and distortion is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to an audio noise reduction model training method, device and storage medium. BACKGROUND

[0002] Audio noise reduction refers to a technology of extracting useful audio signals from noise background when audio signals are disturbed or even submerged by various noises, and suppressing or reducing noise interference. The environment in which we live is full of noise everywhere, so audio signals generally contain different degrees of noise; for example, audio communication in places such as roads, parks and squares will interfere with communication to varying degrees. In order to obtain better audio quality, it is necessary to reduce noise of the audio signal, which essentially extracts as pure original audio as possible from the noisy audio, suppresses various interference signals effectively, enhances the target speech signal, and thus improves the intelligibility of the audio.

[0003] In the prior art, traditional audio noise reduction methods include spectral subtraction, wavelet analysis, signal subspace method, adaptive filter method, etc. However, the traditional audio noise reduction method can usually better suppress the stationary noise scene, but in actual life, the types of noise are various and complex, and the devices for collecting audio are also various, resulting in different noisy audio. The traditional audio enhancement method is based on some set prior assumptions, but these prior assumptions have certain unreasonable places, so the traditional audio is not so good in universality, and in order to adapt to different noise conditions, it relies on parameter setting, manual experience, etc. Time-consuming and laborious parameter adjustment, which brings great challenge to the noise reduction algorithm. In addition, the traditional song noise reduction method is easy to introduce distortion to the original human voice signal in the noise reduction process. SUMMARY

[0004] Therefore, the purpose of the present application is to provide an audio noise reduction model training method, device and medium, which can improve the ability of audio noise reduction. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses an audio noise reduction model training method, comprising:

[0006] Obtain noisy audio, input the noisy audio into a target neural network for training with clean audio as the training target; the target neural network takes noisy audio as input and takes de-noised audio as output;

[0007] Determine the signal-to-noise ratio of the de-noised audio corresponding to the noisy audio as a first signal-to-noise ratio using a signal-to-noise ratio estimator;

[0008] According to the first signal-to-noise ratio and the second signal-to-noise ratio corresponding to the noisy audio, a first loss function is determined;

[0009] The first loss function and the second loss function are used to iteratively train the target neural network to obtain a trained audio noise reduction model; the second loss function is constructed according to the difference between the clean audio corresponding to the noisy audio and the noise-reduced audio.

[0010] Optionally, the signal-to-noise ratio of the noise-reduced audio corresponding to the noisy audio is determined by the signal-to-noise ratio estimator as a first signal-to-noise ratio, comprising:

[0011] According to the spectral feature output by the decoder in the target neural network, the amplitude spectrum corresponding to the noise-reduced audio is determined;

[0012] The amplitude spectrum is input into the signal-to-noise ratio estimator, and the first signal-to-noise ratio is obtained according to the output of the signal-to-noise ratio estimator.

[0013] Optionally, the signal-to-noise ratio estimator is constructed in the order of a double-layer bidirectional long short-term memory network, a one-dimensional convolutional neural network, and an activation function layer, and the signal-to-noise ratio estimator takes the amplitude spectrum as input and outputs the normalized signal-to-noise ratio.

[0014] Optionally, the first loss function is determined according to the first signal-to-noise ratio and a second signal-to-noise ratio corresponding to the noisy audio, comprising:

[0015] A signal-to-noise ratio sequence corresponding to the noisy audio is obtained, and the normalized signal-to-noise ratio corresponding to the signal-to-noise ratio sequence is calculated by using the moving average method combined with the average function, the variance function and the Gaussian error function, as the second signal-to-noise ratio;

[0016] The first loss function for spectral difference is obtained according to the mean square error of the first signal-to-noise ratio and the second signal-to-noise ratio.

[0017] Optionally, the target neural network is constructed in the order of an encoder, a noise reducer and a decoder, and the noisy audio is input into the target neural network for training, comprising:

[0018] The Fourier transform is performed on the noisy audio to obtain a spectral feature corresponding to the noisy audio;

[0019] The spectral feature is input into an encoder in the target neural network to obtain a spectral coding feature output by the encoder;

[0020] The spectral coding feature is input into a noise reducer in the target neural network, and the spectral coding feature is processed by a mask sequence to generate a noise-reduced spectral coding feature;

[0021] The noise-reduced spectral coding feature is input into a decoder in the target neural network to obtain a noise-reduced spectral feature;

[0022] performing Fourier inverse transform on the noise-reduced spectral feature to obtain a noise-reduced audio corresponding to the noisy audio.

[0023] Optionally, the Fourier transform on the noisy audio to obtain a spectral feature corresponding to the noisy audio comprises:

[0024] performing audio framing on the noisy audio to obtain a framed audio signal, and performing Fourier transform on the framed audio signal to obtain a complex domain expression of the noisy audio;

[0025] calculating the amplitude feature and the phase feature from the complex domain expression to obtain the spectral feature.

[0026] Optionally, the obtaining of the noisy audio comprises:

[0027] obtaining noise audios collected by different types of collection devices respectively from different types of noises;

[0028] generating the noisy audio based on the clean audio and the noise audios.

[0029] Optionally, the generating of the noisy audio based on the clean audio and the noise audios comprises:

[0030] performing synthesis processing on the clean audio and the noise audios according to a preset signal-to-noise ratio set to obtain a plurality of corresponding noisy audios.

[0031] In another aspect, the present application discloses an electronic device, comprising:

[0032] a memory for saving a computer program;

[0033] a processor for executing the computer program to implement the foregoing audio noise reduction model training method.

[0034] In another aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the foregoing audio noise reduction model training method.

[0035] In the present application, noisy audio is obtained, and the noisy audio is input into a target neural network with clean audio as a training target for training; the target neural network takes the noisy audio as input and takes denoised audio as output; a signal-to-noise ratio estimator is used to determine the signal-to-noise ratio of the denoised audio corresponding to the noisy audio as a first signal-to-noise ratio; a first loss function is determined according to the first signal-to-noise ratio and a second signal-to-noise ratio corresponding to the noisy audio; the target neural network is iteratively trained using the first loss function and a second loss function, to obtain a trained audio denoising model; the second loss function is constructed according to the difference between the clean audio corresponding to the noisy audio and the denoised audio. As can be seen, the target neural network is trained using the noisy audio, the signal-to-noise ratio of the denoised audio corresponding to the noisy audio is estimated using a signal-to-noise ratio estimator, and the first loss function constructed according to the signal-to-noise ratio difference between the noisy audio and the denoised audio, and the second loss function constructed according to the difference between the clean audio corresponding to the noisy audio and the denoised audio are used for training constraint in the training process. After multiple iterations, a trained audio denoising model is obtained. Thus, the audio denoising model based on the deep neural network has good generalization ability after a large amount of training, and can better remove noise for steady-state noise and more complex and variable background noise. The first loss function constructed according to the signal-to-noise ratio difference is used for constraint to avoid excessive inhibition, the fidelity of the human voice signal is better, distortion is reduced, and the audio denoising capability is improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0037] Figure 1 A flow chart of an audio denoising model training method is provided for the present application;

[0038] Figure 2 A specific audio framing schematic diagram is provided for the present application;

[0039] Figure 3 A specific denoiser learning flow chart is provided for the present application;

[0040] Figure 4 A specific audio denoising system block diagram is provided for the present application;

[0041] Figure 5 A flow chart of a specific audio denoising model training method is provided for the present application;

[0042] Figure 6 An audio noise reduction method flow chart is provided for the present application;

[0043] Figure 7 A specific noise reduction prediction flow chart is provided for the present application;

[0044] Figure 8 An audio noise reduction model training device structure schematic diagram is provided for the present application;

[0045] Figure 9 An electronic device structure diagram is provided for the present application;

[0046] Figure 10 An audio noise reduction model training method and / or hardware composition framework schematic diagram applicable to the audio noise reduction method are provided for the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0048] In the prior art, traditional audio noise reduction methods include spectral subtraction, wavelet analysis, signal subspace method, adaptive filter method, etc. However, the traditional audio noise reduction method can usually better suppress the stationary noise scene, but in actual life, the types of noise are various and complex, and the devices for collecting audio are also various, resulting in that the collected noisy audio is also different. The traditional audio enhancement method is based on some set prior assumptions, but these prior assumptions have certain unreasonable places, so the traditional audio is not so good in universality, and in order to adapt to different noise conditions, it depends on the setting of parameters, manual experience, etc., and the complicated parameter adjustment is time-consuming and laborious, which brings great challenge to the noise reduction algorithm. In addition, the traditional song noise reduction method is easy to introduce distortion to the original human voice signal in the noise reduction process. In order to overcome the above-described problems, the present application provides an audio noise reduction model training method, which can improve the ability of audio noise reduction, and at the same time, the fidelity of human voice signal is better and the distortion is reduced.

[0049] The embodiments of the present application disclose an audio noise reduction model training method, as shown in Figure 1 The method can include the following steps:

[0050] Step S11: obtaining noisy audio, inputting the noisy audio into a target neural network for training with clean audio as the training target; the target neural network takes noisy audio as input and outputs denoised audio.

[0051] In this embodiment, first, the noisy audio is obtained, and then the noisy audio is input into the target neural network, and the network is trained with clean audio as the training target. The target neural network is a neural network that takes noisy audio as input and outputs denoised audio. The target neural network can be a neural network with a structure of CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), or DNN (Deep Neural Network).

[0052] In this embodiment, the obtaining of the noisy audio includes: obtaining noise audio collected by different types of collection devices; and generating noisy audio based on clean audio and the noise audio. That is, a large amount of clean audio and various types of noise audio are collected, such as noise types of square, road, conference room, restaurant, coffee shop, keyboard tapping sound, etc., and then various types of noisy audio are generated based on clean audio and various types of noise audio.

[0053] In this embodiment, the generating of the noisy audio based on the clean audio and the noise audio includes: synthesizing the clean audio and the noise audio according to a preset signal-to-noise ratio set to obtain a plurality of corresponding noisy audio. That is, the clean audio and the noise audio are mixed to obtain the noisy audio, which can be synthesized according to different signal-to-noise ratios to meet different noise level scenarios. The signal-to-noise ratio refers to the ratio of signal to noise, such as a signal-to-noise ratio of -15 to 20 dB, which is not limited in this embodiment and can be selected according to actual demand scenarios.

[0054] In this embodiment, the target neural network can be constructed in the order of an encoder, a denoiser, and a decoder, that is, the denoised audio input into the target neural network is sequentially output as corresponding denoised audio after passing through the encoder, the denoiser, and the decoder. The inputting of the noisy audio into the target neural network for training can include the following steps:

[0055] S111: performing Fourier transform on the noisy audio to obtain a frequency spectrum feature corresponding to the noisy audio;

[0056] That is, the Fourier transform is first performed on the noisy audio to obtain the frequency spectrum feature of the noisy audio, and then the feature is input into the target neural network for training.

[0057] In this embodiment, the Fourier transform of the noisy audio can include: performing audio framing on the noisy audio to obtain a framed audio signal, performing Fourier transform on the framed audio signal to obtain a complex domain expression of the noisy audio; and calculating the amplitude feature and the phase feature corresponding to the noisy audio according to the complex domain expression to obtain the frequency spectrum feature. That is, the Fourier transform can be used to extract features of the noisy audio, convert from the time domain waveform to the time domain, obtain the amplitude feature and the phase feature of the noisy audio, and obtain the frequency spectrum feature based on the amplitude feature and the phase feature.

[0058] As shown in Figure 2 , the audio is framed with a frame shift of 10 ms and a frame length of 20 ms. If it is assumed that the audio data to be denoised is represented as x ∈ R 1×T , where T is the number of sampling points. First, the input waveform x is framed, where the frame length is L (generally an exponential multiple of 2, such as 1024) and the frame shift is P (such as 0.5L, such as Figure 2 ). A total of C frames are divided. After the framed signal is subjected to Fourier transform, a total of L frequency points are obtained. Due to the symmetric conjugate property of the frequency points, generally L / 2+1 frequency points are taken. For convenience of representation, it is assumed that P frequency points are taken. Therefore, the real part and the imaginary part of the feature signal after the Fourier transform are respectively X r ∈ R P×C and X i ∈ R P×C . Therefore, the real part and the imaginary part parameters in the encoder can be defined as two-dimensional matrices E r ∈ R P×N and E i ∈ R P×N . Correspondingly, the real part and the imaginary part parameters in the decoder can be defined as two-dimensional matrices D r ∈ R N×P and D i ∈ R N×P , where N represents the feature dimension of the encoder and the decoder, which can be self-defined, such as N being 128 or 256. Since the encoder, the decoder and the like have real parts and imaginary parts, for the convenience of description, only the real part is described, and the imaginary part can be omitted. The two-dimensional matrix E of the encoder described above actually represents a real part two-dimensional matrix and an imaginary part two-dimensional matrix. It is assumed that the result of the Fourier transform of the audio is a complex domain: X = X r +iX i , the corresponding amplitude is: , and the corresponding phase is: α = arctan(X i / X rHere, arctan is the arctangent function. This involves performing a short-time Fourier transform on the time-domain waveform of the noise-reducing frequency signal to obtain the transformed initial spectrum; where |X| is the amplitude characteristic of the extracted initial spectrum, and α is the phase characteristic of its initial spectrum.

[0059] S112: Input the spectral features into the encoder in the target neural network to obtain the spectral coding features output by the encoder;

[0060] In this embodiment, the obtained spectral features are input into the encoder in the target neural network to obtain the spectral encoded features output by the encoder. It can be understood that the encoder is a two-dimensional matrix E∈R. P×N The encoded output is F∈R N×C That is, F = E T X, where T represents transpose. The encoder calculates the spectral characteristics of the noise-reducing frequency signal to be denoised, thereby generating the corresponding spectral coding features.

[0061] S113: Input the spectrum coding features into the denoiser in the target neural network, and use the mask sequence to denoise the spectrum coding features to generate denoised spectrum coding features;

[0062] In this embodiment, the spectral coding features output by the encoder are input to the noise reduction unit. That is, the noise reduction unit input is the real part sequence and the imaginary part sequence after the encoder, and the output is the real part masking sequence and the imaginary part masking sequence, denoted as M = M r +iM i .For example Figure 3 As shown, the real and imaginary part sequences output by the encoder are input into the real part processing network to obtain the real part mask sequence, and the real and imaginary part sequences output by the encoder are input into the imaginary part processing network to obtain the imaginary part mask sequence. Masking is achieved by learning the probability of speech and noise at each time-frequency point through a neural network. Generally, the masking matrix is ​​a two-dimensional matrix of 0 to 1; the closer it is to 1, the higher the probability of speech. Thus, by performing a dot product calculation, the speech signal can be preserved, achieving the purpose of noise reduction. The noise reduction output is S = S r +iS i It includes the real part and the imaginary part: S r =F r *M r -F i *M i S i =F r *M i +F i *M rThe mask sequence calculates the spectral coding features of the to-be-noise-reduced audio signal, thereby generating noise-reduced spectral coding features, i.e., spectral coding features of a clean signal.

[0063] In the embodiment, the noise-reduced spectral coding features output by the noise reducer are input into a decoder in the target neural network, thereby obtaining noise-reduced spectral features.

[0064] In the embodiment, the noise-reduced spectral coding features output by the noise reducer are input into a decoder, and the decoder outputs noise-reduced spectral features after decoding. It can be understood that the decoder is a two-dimensional matrix D∈R N×P The input of the decoder is the frequency domain feature after the noise reducer, and the output result after decoding is V∈R P×C , i.e., V=D T S, where T represents transposition. The decoder calculates the noise-reduced spectral coding features, thereby generating corresponding noise-reduced spectral features.

[0065] S115: performing inverse Fourier transform on the noise-reduced spectral features, thereby obtaining the noise-reduced audio corresponding to the noisy audio.

[0066] In the embodiment, when the noise-reduced spectral features output by the decoder, the corresponding amplitude and phase can be obtained through the real part and the imaginary part, and the enhanced frequency domain spectrum is further obtained based on the amplitude and the phase. That is, according to the noise-reduced spectral features, the amplitude features, and the phase features, the target complex spectrum can be calculated. Then, the short-time inverse Fourier transform is performed on the target complex spectrum: Z=Z r +iZ i =|V|cos(β)+i|V|sin(β), thereby performing waveform reconstruction, i.e., obtaining the reconstructed noise-reduced audio.

[0067] Step S12: determining the signal-to-noise ratio of the noise-reduced audio corresponding to the noisy audio as the first signal-to-noise ratio by using the signal-to-noise ratio estimator.

[0068] In the embodiment, the signal-to-noise ratio of the noise-reduced audio corresponding to the noisy audio is determined as the first signal-to-noise ratio by using the signal-to-noise ratio estimator, i.e., the signal-to-noise ratio of the noise-reduced audio is estimated by using the signal-to-noise ratio estimator based on the noise-reduced audio, and the signal-to-noise ratio of the noise-reduced audio is taken as the first signal-to-noise ratio.

[0069] In the embodiment, the determination of the signal-to-noise ratio of the noise-reduced audio corresponding to the noisy audio by the signal-to-noise ratio estimator as the first signal-to-noise ratio can include: determining an amplitude spectrum corresponding to the noise-reduced audio according to the noise-reduced spectral feature output by the target neural network decoder; and inputting the amplitude spectrum into the signal-to-noise ratio estimator to obtain the first signal-to-noise ratio according to an output of the signal-to-noise ratio estimator. It can be understood that the noise-reduced spectral feature output by the target neural network decoder includes an amplitude spectrum and a phase spectrum, and the amplitude spectrum is extracted and input into the signal-to-noise ratio estimator to obtain the first signal-to-noise ratio according to the output of the signal-to-noise ratio estimator.

[0070] In the embodiment, the signal-to-noise ratio estimator is constructed in the order of a two-layer bidirectional long short-term memory network, a one-dimensional convolutional neural network, and an activation function layer, and the signal-to-noise ratio estimator takes the amplitude spectrum as input and outputs a normalized signal-to-noise ratio. That is, the signal-to-noise ratio estimator adopts a two-layer bidirectional LSTM (Long Short-Term Memory) module, then interfaces a one-dimensional CNN convolution, and takes a sigmoid function as an activation function, which can map a real number to the interval (0, 1), that is, the normalized snr (Signal to Interference plus Noise Ratio) predicts the phase β of the noise-reduced spectral feature: β = arctan(V i / V r ), and the amplitude: The input of the signal-to-noise ratio estimator is the amplitude spectrum of the decoder output feature, that is, |V|, and the output is the normalized signal-to-noise ratio, that is, the first signal-to-noise ratio, which is denoted as snr_predict.

[0071] Step S13: determining a first loss function according to the first signal-to-noise ratio and a second signal-to-noise ratio corresponding to the noisy audio.

[0072] In the embodiment, the first loss function is constructed according to a difference between the second signal-to-noise ratio corresponding to the noisy audio and the first signal-to-noise ratio corresponding to the noise-reduced audio, and is a loss function for spectral difference.

[0073] In this embodiment, the first loss function is determined according to the first signal-to-noise ratio and the second signal-to-noise ratio corresponding to the noisy audio. The method can include: obtaining a signal-to-noise ratio sequence corresponding to the noisy audio, calculating a normalized signal-to-noise ratio corresponding to the signal-to-noise ratio sequence by using a sliding average method combined with a mean value function, a variance function and a Gaussian error function, and taking the normalized signal-to-noise ratio as the second signal-to-noise ratio; and obtaining the first loss function for spectral difference according to the mean square error of the first signal-to-noise ratio and the second signal-to-noise ratio. That is, assuming that the signal-to-noise ratio sequence corresponding to the noisy audio is snr, then the normalized signal-to-noise ratio corresponding to the signal-to-noise ratio sequence, that is, the normalization step of the true value snr, is as follows:

[0074] The initial values of snr_mean and snr_std are 0, and the sliding average is used for iterative calculation to avoid excessive fluctuation of the value of snr. Wherein alpha takes a value close to 1, such as 0.99 or 0.98. Wherein mean() refers to the mean value function, std() refers to the variance function, and erf() refers to the error function.

[0075] snr_mean = snr_mean * alpha + mean(snr) * (1-alpha);

[0076] snr_std = snr_std * alpha + std(snr) * (1-alpha);

[0077] snr1 = (snr-snr_mean) / snr_std;

[0078] snr_norm = (erf(snr1) + 1) / 2;

[0079] The first loss function is the loss of snr, denoted as snr_loss, and snr_loss takes the mean square error of snr_predict and snr_norm.

[0080] Step S14: iteratively training the target neural network by using the first loss function and the second loss function to obtain an audio denoising model after training.

[0081] The second loss function is constructed according to the difference between the clean audio corresponding to the noisy audio and the denoised audio. For example Figure 4As shown, a loss function is used for constraints during the training process. In this embodiment, the target neural network can be iteratively trained using both a first loss function and a second loss function to obtain the trained audio noise reduction model. The first loss function is constructed based on the difference between the second signal-to-noise ratio corresponding to the noisy frequency and the first signal-to-noise ratio corresponding to the noise-reduced frequency, i.e., it constrains and guides training based on the difference in signal-to-noise ratio. The second loss function is constructed based on the difference between the clean audio corresponding to the noisy frequency and the noise-reduced frequency. The accuracy of training is improved through joint constraints.

[0082] In this embodiment, training is performed on a noisy frequency signal, with the corresponding clean audio signal as the target, and constrained by a loss function. Let the noisy frequency signal be x, the clean audio signal be y, and the predicted output audio signal of the denoising model be z. Fourier transforms are performed on each signal, resulting in spectral distributions Xf, Yf, and Zf. The errors in Yf and Zf are calculated, and then the hyperparameters in the model are adjusted based on these errors, i.e., constrained by the second loss function, to ensure the weights in the model are sufficiently trained and learned. Through continuous iteration, training is completed when the error is less than a preset value, resulting in the trained preset neural network model.

[0083] For example Figure 5 As shown, by pairing clean and noisy signals, with the clean signal as the target, the mean square error between the predicted waveform signal and the true clean signal is used as the second loss function, denoted as wav_loss. Figure 5 Loss function 2. The difference between the second signal-to-noise ratio (SNR) corresponding to the noisy frequency and the first SNR corresponding to the denoised frequency is used as the first loss function, denoted as snr_loss, i.e. Figure 5 The loss function is 1, and the total loss function is snr_loss + wav_loss. It can be seen that the end-to-end noise reduction system uses an encoder, denoiser, and decoder structure. The encoder mainly transforms the input audio signal to obtain richer features. The denoiser receives the features generated by the encoder and then performs noise reduction. The decoder reconstructs the waveform signal from the output of the denoiser as the enhanced signal after noise reduction. A signal-to-noise ratio (SNR) estimator is specifically introduced to align with the original added SNR, avoiding over-suppression and the resulting signal distortion.

[0084] As can be seen, in the embodiment, noisy audio is acquired, clean audio is taken as a training target, and the noisy audio is input into a target neural network for training; the target neural network takes the noisy audio as input and takes denoised audio as output; a signal-to-noise ratio estimator is used to determine a signal-to-noise ratio of the denoised audio corresponding to the noisy audio as a first signal-to-noise ratio; a first loss function is determined according to the first signal-to-noise ratio and a second signal-to-noise ratio corresponding to the noisy audio; the target neural network is iteratively trained by using the first loss function and a second loss function, to obtain a trained audio denoising model; the second loss function is constructed according to a difference between clean audio corresponding to the noisy audio and the denoised audio. As can be seen, the target neural network is trained by using the noisy audio, the signal-to-noise ratio of the denoised audio corresponding to the noisy audio is estimated by using the signal-to-noise ratio estimator, and the first loss function constructed according to the signal-to-noise ratio difference between the noisy audio and the denoised audio and the second loss function constructed according to the difference between the clean audio corresponding to the noisy audio and the denoised audio are used for training constraint in the training process, and the trained audio denoising model is obtained after multiple iterations. Thus, the audio denoising model based on the deep neural network has good generalization ability after a large amount of training, can better remove noise for steady-state noise and more complex and variable background noise, is constrained by the first loss function constructed according to the signal-to-noise ratio difference, avoids excessive inhibition, is better faithful to a human voice signal, reduces distortion, and improves the audio denoising capability.

[0085] The embodiment of the present application discloses an audio denoising method, as shown in Figure 6 The method can include the following steps:

[0086] Step S21: acquiring audio to be denoised.

[0087] Step S22: inputting the audio to be denoised into the trained audio denoising model to obtain denoised audio corresponding to the audio to be denoised and output by the trained audio denoising model.

[0088] The trained audio denoising model is obtained by training a target neural network by using noisy audio and iteratively training the target neural network by using a first loss function and a second loss function in a training process; the target neural network is constructed in the order of an encoder, a denoiser, and a decoder, takes noisy audio as input and takes denoised audio as output; the first loss function is constructed according to a difference between a second signal-to-noise ratio corresponding to the noisy audio and a first signal-to-noise ratio corresponding to the denoised audio, and the second loss function is constructed according to a difference between clean audio corresponding to the noisy audio and the denoised audio.

[0089] In this embodiment, the audio to be de-noised is input into the trained audio de-noising model, and the specific process includes: performing Fourier transform on the audio to be de-noised to obtain the frequency spectrum features corresponding to the audio to be de-noised; inputting the frequency spectrum features into an encoder in the target neural network to obtain the frequency spectrum coding features output by the encoder; inputting the frequency spectrum coding features into a de-noiser in the target neural network, performing de-noising processing on the frequency spectrum coding features by using a mask sequence to generate de-noised frequency spectrum coding features; inputting the de-noised frequency spectrum coding features into a decoder in the target neural network to obtain de-noised frequency spectrum features; and performing inverse Fourier transform on the de-noised frequency spectrum features to obtain the de-noised audio corresponding to the audio to be de-noised.

[0090] In this embodiment, the Fourier transform on the audio to be de-noised to obtain the frequency spectrum features corresponding to the audio to be de-noised includes: performing audio framing on the audio to be de-noised to obtain a framed audio signal, performing Fourier transform on the framed audio signal to obtain a complex domain expression of the audio to be de-noised; and calculating the amplitude features and the phase features corresponding to the audio to be de-noised according to the complex domain expression to obtain the frequency spectrum features of the audio to be de-noised.

[0091] For example Figure 7 As shown in the de-noising prediction flowchart, the de-noising prediction process is generally close to the training process. The noisy audio is subjected to feature extraction, and then the trained de-noising model is used for prediction to output the predicted audio frequency spectrum features. Then, the de-noised audio signal can be obtained through parallel reconstruction, i.e., inverse Fourier transform, and the calculation steps are the same as those in the training process. Finally, the noisy audio signal is subjected to appropriate de-noising to obtain a cleaner audio signal, so that the intelligibility and the like are improved.

[0092] As can be seen, in the embodiment, noisy audio is acquired, the noisy audio is input into a target neural network for training with clean audio as a training target; the target neural network is a neural network constructed in the order of an encoder, a noise reducer and a decoder, with noisy audio as input and with denoised audio as output; the target neural network is iteratively trained by using a first loss function and a second loss function, to obtain a trained audio noise reduction model; the first loss function is constructed according to the difference between the second signal-to-noise ratio corresponding to the noisy audio and the first signal-to-noise ratio corresponding to the denoised audio, and the second loss function is constructed according to the difference between the clean audio corresponding to the noisy audio and the denoised audio. In this way, the target neural network is trained by using noisy audio, and the first loss function constructed according to the signal-to-noise ratio difference between the noisy audio and the denoised audio and the second loss function constructed according to the difference between the clean audio corresponding to the noisy audio and the denoised audio are used for training constraint in the training process, and the trained audio noise reduction model is obtained after multiple iterations. The audio noise reduction model based on the deep neural network has good generalization ability after a large amount of training, and can better remove noise for steady-state noise and more complex and variable background noise. The first loss function constructed according to the signal-to-noise ratio difference is used for constraint to avoid excessive inhibition and better preserve the fidelity of the human voice signal, reduce distortion, and improve the audio noise reduction capability.

[0093] Next, taking a music APP as an example, the technical solutions of the present application are described.

[0094] At present, when using a K song APP on the market to record, due to the limitation of non-professional equipment and environment, noise (microphone friction sound, environmental background noise, etc.) is easily mixed into the song recorded by the user, which has a great influence on the listening experience. Therefore, it is necessary to reduce the noise of the recorded song. Similarly, in the voice call scene, there is also a similar noise problem. The background server of the music APP acquires noisy audio, inputs the noisy audio into a target neural network for training with clean audio as a training target; the target neural network is a neural network constructed in the order of an encoder, a noise reducer and a decoder, with noisy audio as input and with denoised audio as output; the target neural network is iteratively trained by using a first loss function constructed according to the difference between the second signal-to-noise ratio corresponding to the noisy audio and the first signal-to-noise ratio corresponding to the denoised audio, to obtain a trained audio noise reduction model. After training, the user uploads the audio to be denoised, and inputs the audio to be denoised into the trained audio noise reduction model to obtain the denoised audio corresponding to the audio to be denoised output by the trained audio noise reduction model.

[0095] Correspondingly, the embodiment of the present application also discloses an audio noise reduction model training device, as shown in Figure 8 The device comprises:

[0096] The noisy audio acquisition module 11 is configured to acquire noisy audio, input the noisy audio into a target neural network with clean audio as a training target for training, and input the noisy audio into the target neural network with the denoised audio as output;

[0097] The first signal-to-noise ratio determination module 12 is configured to determine a signal-to-noise ratio of the denoised audio corresponding to the noisy audio as a first signal-to-noise ratio by using a signal-to-noise ratio estimator.

[0098] The first loss function determination module 13 is configured to determine a first loss function according to the first signal-to-noise ratio and a second signal-to-noise ratio corresponding to the noisy audio.

[0099] The training module 14 is configured to perform iterative training on the target neural network by using the first loss function and a second loss function, and obtain a trained audio denoising model; the second loss function is constructed according to a difference between the clean audio corresponding to the noisy audio and the denoised audio.

[0100] As can be seen from the above, in the embodiment, the noisy audio is acquired, the noisy audio is input into the target neural network with the clean audio as a training target for training, the target neural network inputs the noisy audio and outputs the denoised audio, the signal-to-noise ratio of the denoised audio corresponding to the noisy audio is determined as a first signal-to-noise ratio by using a signal-to-noise ratio estimator, a first loss function is determined according to the first signal-to-noise ratio and a second signal-to-noise ratio corresponding to the noisy audio, the target neural network is iteratively trained by using the first loss function and a second loss function, and a trained audio denoising model is obtained; the second loss function is constructed according to a difference between the clean audio corresponding to the noisy audio and the denoised audio. As can be seen, the target neural network is trained by using the noisy audio, the signal-to-noise ratio of the denoised audio corresponding to the noisy audio is estimated by using the signal-to-noise ratio estimator, and the first loss function constructed according to the signal-to-noise ratio difference between the noisy audio and the denoised audio and the second loss function constructed according to the difference between the clean audio corresponding to the noisy audio and the denoised audio are used for training constraint in the training process, and the trained audio denoising model is obtained after multiple iterations. Thus, the audio denoising model based on the deep neural network has good generalization ability after a large amount of training, can better remove noise for the steady-state noise and the more complex and variable background noise of the non-steady-state noise, is constrained by the first loss function constructed according to the signal-to-noise ratio difference, avoids excessive inhibition, is better faithful to the human voice signal, reduces distortion, and improves the audio denoising capability.

[0101] Further, the embodiment of the present application also discloses an electronic device, as shown in Figure 9As shown in the figure, a hardware component framework diagram applicable to the audio noise reduction model training method and / or the audio noise reduction method provided by the embodiments of the present application is shown. The content in the figure should not be considered as any limitation on the use range of the present application.

[0102] Figure 9 A structural diagram of an electronic device 20 provided by the embodiments of the present application is shown. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the related steps in the audio noise reduction model training method disclosed in any of the preceding embodiments.

[0103] In the embodiments, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0104] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc., and the resources stored thereon include an operating system 221, a computer program 222, and data 223 including noisy audio, etc., and the storage mode can be temporary storage or permanent storage.

[0105] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the processor 21 on the mass data 223 in the memory 22, and the operating system 221 can be Windows Server, Netware, Unix, Linux, etc. The computer program 222 can further include computer programs for completing other specific work in addition to the computer programs for completing the audio noise reduction model training method executed by the electronic device 20 disclosed in any of the preceding embodiments.

[0106] It can be understood that the number of electronic devices in the embodiments of the present application is not limited, and multiple electronic devices can be used to cooperatively complete the audio noise reduction model training method and / or the audio noise reduction method. In one possible implementation, please refer to Figure 10 , Figure 10 A hardware component framework diagram applicable to another audio noise reduction model training method and / or audio noise reduction method provided by the embodiments of the present application is shown.Figure 10 It can be known that the hardware composition framework can include a first electronic device 31 and a second electronic device 32 connected through a network 33.

[0107] In the embodiments of the present application, the hardware structure of the first electronic device 31 and the second electronic device 32 can refer to the hardware structure of the electronic device in the prior art. Figure 9 That is, it can be understood that there are two electronic devices in the embodiments of the present application, and the two electronic devices interact with each other. Further, the form of the network 33 is not limited in the embodiments of the present application, that is, the network 33 can be a wireless network (such as WIFI, Bluetooth, etc.), or a wired network.

[0108] The first electronic device 31 and the second electronic device 32 can be the same type of electronic device, such as a server. They can also be different types of electronic devices, for example, the first electronic device 31 can be a smart phone or other smart terminal, and the second electronic device 32 can be a server. In a possible implementation, a server with strong computing power can be used as the first electronic device 31 and the second electronic device 32 to improve the model training efficiency. In a possible implementation, a server with strong computing power can be used as the second electronic device 32 to improve the data processing efficiency and reliability, and thus improve the processing efficiency of audio noise reduction. At the same time, a smart phone with low cost and wide application range is used as the first electronic device 31 to realize the interaction between the second electronic device 32 and the user. It can be understood that the interaction process can be that the smart phone sends the audio to be de-noised to the server, the server inputs the audio to be de-noised into the trained audio noise reduction model to obtain the de-noised audio corresponding to the audio to be de-noised output by the trained audio noise reduction model; wherein the trained audio noise reduction model is obtained by training a target neural network with noisy audio and iteratively training the target neural network with a first loss function and a second loss function during the training process; wherein the target neural network is a neural network constructed in the order of an encoder, a noise reducer and a decoder, with noisy audio as input and de-noised audio as output; wherein the first loss function is constructed according to the difference between the second signal-to-noise ratio corresponding to the noisy audio and the first signal-to-noise ratio corresponding to the de-noised audio, and the second loss function is constructed according to the difference between the clean audio corresponding to the noisy audio and the de-noised audio. Finally, the server saves or feeds back the de-noised audio to the smart phone.

[0109] Further, the present application also discloses a computer storage medium, which stores computer executable instructions, and when the computer executable instructions are loaded and executed by a processor, the steps of the audio noise reduction model training method disclosed in any of the preceding embodiments are implemented.

[0110] The various embodiments described in this specification are presented by way of example, and each embodiment is not necessarily composed of all features described with respect to other embodiments. Each embodiment described in this specification can be combined with one or more other embodiments described in this specification, and is not limited to the combinations explicitly enumerated in this specification. Furthermore, the various embodiments described in this specification can be combined with one or more other embodiments described in other specifications.

[0111] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, hard disk can be used as a storage medium.

[0112] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and do not imply singular or plural. Moreover, the terms "include", "have", or any other variant thereof are intended to encompass non-exclusive inclusions, such that processes, methods, articles, or apparatuses that comprise a set of elements not expressly listed are also within the scope of such processes, methods, articles, or apparatuses. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0113] The above describes in detail the audio noise reduction model training method, device, equipment and medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above embodiment descriptions are only used to help understand the method and core idea of the present application. For those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for training an audio noise reduction model, characterized in that, The method comprises: inputting the noisy audio into a target neural network for training, with clean audio as a training target; the target neural network inputs the noisy audio and outputs denoised audio; based on the amplitude spectrum corresponding to the denoised audio, a signal-to-noise ratio estimator is used to determine the signal-to-noise ratio of the denoised audio corresponding to the noisy audio as a first signal-to-noise ratio; the signal-to-noise ratio estimator is constructed in the order of a double-layer bidirectional long short-term memory network, a one-dimensional convolutional neural network, and an activation function layer; determining a first loss function according to the first signal-to-noise ratio and a second signal-to-noise ratio corresponding to the noisy audio; iteratively training the target neural network using the first loss function and the second loss function to obtain a trained audio denoising model; the second loss function is constructed according to the waveform difference between the clean audio corresponding to the noisy audio and the denoised audio.

2. The audio noise reduction model training method of claim 1, wherein, The method comprises: determining the amplitude spectrum corresponding to the denoised audio according to the denoised spectral feature output by the decoder in the target neural network; inputting the amplitude spectrum into the signal-to-noise ratio estimator to obtain the first signal-to-noise ratio according to the output of the signal-to-noise ratio estimator.

3. The audio noise reduction model training method of claim 1, wherein, The signal-to-noise ratio estimator inputs the amplitude spectrum and outputs a normalized signal-to-noise ratio.

4. The audio noise reduction model training method of claim 1, wherein, The method comprises: obtaining a signal-to-noise ratio sequence corresponding to the noisy audio, and calculating the normalized signal-to-noise ratio corresponding to the signal-to-noise ratio sequence using a moving average method combined with an average function, a variance function, and a Gaussian error function as the second signal-to-noise ratio; obtaining the first loss function for spectral difference according to the mean square error of the first signal-to-noise ratio and the second signal-to-noise ratio.

5. The audio noise reduction model training method of claim 1, wherein, The target neural network is constructed in the order of an encoder, a denoiser, and a decoder, and the method comprises: performing Fourier transform on the noisy audio to obtain a spectral feature corresponding to the noisy audio; inputting the spectral feature into the encoder in the target neural network to obtain a spectral coding feature output by the encoder; inputting the spectral coding feature into the denoiser in the target neural network, performing denoising processing on the spectral coding feature using a mask sequence to generate a denoised spectral coding feature; inputting the denoised spectral coding feature into the decoder in the target neural network to obtain a denoised spectral feature; performing inverse Fourier transform on the denoised spectral feature to obtain a denoised audio corresponding to the noisy audio.

6. The audio noise reduction model training method of claim 5, wherein, The method comprises: performing audio framing on the noisy audio to obtain a framed audio signal, and performing Fourier transform on the framed audio signal to obtain a complex domain expression of the noisy audio; calculating the amplitude feature and the phase feature corresponding to the noisy audio according to the complex domain expression to obtain the spectral feature.

7. The audio noise reduction model training method of any one of claims 1-6, wherein, The method comprises: acquire noise audios obtained by collecting different types of noises through different types of collecting devices; generate a noisy audio based on the clean audio and the noise audios.

8. The audio noise reduction model training method of claim 7, characterized in that, The generating of the noisy audio based on the clean audio and the noise audios comprises: performing a synthesis processing on the clean audio and the noise audios according to a preset signal-to-noise ratio set to obtain a plurality of corresponding noisy audios.

9. An electronic device, comprising: comprise: a memory for saving a computer program; a processor for executing the computer program to implement the audio noise reduction model training method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, a memory for storing a computer program; wherein the computer program is executed by a processor to implement the audio noise reduction model training method according to any one of claims 1 to 8.

Images