Noise reduction model training method, device and apparatus

Abstract

The present disclosure provides a noise reduction model training method, device and equipment, relates to the technical field of artificial intelligence, in particular to the technical field of natural language processing and deep learning. A specific embodiment of the method comprises: obtaining a sample noisy speech signal; extracting a sample frequency domain feature of the sample noisy speech signal; calculating a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal; training a network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision, to obtain a noise reduction model. The noise reduction model trained by the embodiment has good noise reduction performance.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of natural language processing and deep learning. BACKGROUND

[0002] Real-time voice calls can bring great convenience to people's life and work. However, with the continuous development of technology and life, people's requirements for the quality of voice calls are also getting higher and higher. In the face of various noisy product application environments, voice noise reduction technology plays an irreplaceable role, which can suppress background noise in voice signals, improve the intelligibility and intelligibility of voice, and thus improve the quality and efficiency of voice interaction. In real-time calls, its noise suppression capability and real-time running capability play a crucial role.

[0003] Currently, classic noise reduction algorithms such as spectral subtraction and adaptive filter noise reduction are used for voice noise reduction. SUMMARY

[0004] The embodiments of the present disclosure provide a noise reduction model training method, device, equipment, storage medium and program product.

[0005] In a first aspect, the embodiments of the present disclosure provide a noise reduction model training method, comprising: obtaining a sample noisy speech signal; extracting a sample frequency domain feature of the sample noisy speech signal; calculating a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal; training a network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision, to obtain a noise reduction model.

[0006] In a second aspect, the embodiments of the present disclosure provide a voice noise reduction method, comprising: obtaining a noisy speech signal; extracting a frequency domain feature of the noisy speech signal; inputting the frequency domain feature into a noise reduction model to obtain M frequency band gains, wherein M is a positive integer, and the noise reduction model is trained by the method of the first aspect; and performing noise reduction based on the M frequency band gains to obtain a clean speech signal.

[0007] In a third aspect, the embodiments of the present disclosure provide a noise reduction model training device, comprising: an obtaining module configured to obtain a sample noisy speech signal; an extracting module configured to extract a sample frequency domain feature of the sample noisy speech signal; a calculating module configured to calculate a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal; and a training module configured to train a network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision, to obtain a noise reduction model.

[0008] In a fourth aspect, the embodiments of the present disclosure provide a speech noise reduction device, comprising: an acquisition module configured to acquire a noisy speech signal; an extraction module configured to extract a frequency domain feature of the noisy speech signal; an input module configured to input the frequency domain feature into a noise reduction model to obtain M frequency band gains, wherein M is a positive integer, and the noise reduction model is trained by the device of the third aspect; and a noise reduction module configured to perform noise reduction based on the M frequency band gains to obtain a clean speech signal.

[0009] In a fifth aspect, the embodiments of the present disclosure provide an electronic device, comprising: at least two processors; and a memory in communication connection with the at least two processors; wherein the memory stores instructions executable by the at least two processors, and the instructions are executed by the at least two processors to enable the at least two processors to perform the method described in any one of the implementations of the first aspect or the method described in any one of the implementations of the second aspect.

[0010] In a sixth aspect, the embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to make a computer execute the method described in any one of the implementations of the first aspect or the method described in any one of the implementations of the second aspect.

[0011] In a seventh aspect, the embodiments of the present disclosure provide a computer program product, comprising a computer program, the computer program being used to implement the method described in any one of the implementations of the first aspect or the method described in any one of the implementations of the second aspect when executed by a processor.

[0012] The noise reduction model training method provided by the embodiments of the present disclosure has good noise reduction performance and real-time performance, and has good suppression ability for stationary noise and non-stationary noise.

[0013] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0014] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings. The drawings are intended to better understand the present scheme and do not constitute a limitation of the present disclosure. Among them:

[0015] Figure 1 is a flowchart of one embodiment of the noise reduction model training method according to the present disclosure;

[0016] Figure 2 is a flowchart of another embodiment of the noise reduction model training method according to the present disclosure;

[0017] Figure 3 is Figure 2 a schematic diagram of a training feature in the denoising model training method in

[0018] Figure 4 is Figure 2 a network structure diagram of a denoising model in the denoising model training method in

[0019] Figure 5 is a flowchart of one embodiment of a voice denoising method according to the present disclosure;

[0020] Figure 6 is a flowchart of another embodiment of a voice denoising method according to the present disclosure;

[0021] Figure 7 is Figure 6 a flowchart of a voice denoising method in

[0022] Figure 8 is a denoising effect schematic diagram;

[0023] Figure 9 is another denoising effect schematic diagram;

[0024] Figure 10 is a scene diagram in which the denoising model training method and the voice denoising method according to the embodiments of the present disclosure can be implemented;

[0025] Figure 11 is a structural schematic diagram of one embodiment of a denoising model training device according to the present disclosure;

[0026] Figure 12 is a structural schematic diagram of one embodiment of a voice denoising device according to the present disclosure;

[0027] Figure 13 is a block diagram of an electronic device for implementing the denoising model training method and the voice denoising method according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0029] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] Figure 1 Flow 100 of one embodiment of the noise reduction model training method according to the present disclosure is shown. The noise reduction model training method includes the following steps:

[0031] Step 101, obtaining a sample noisy speech signal.

[0032] In this embodiment, the execution subject of the noise reduction model training method can obtain a sample noisy speech signal. The sample noisy speech signal is a speech signal with noise, which can be a speech signal collected in a noisy environment or obtained by fusing a clean speech signal and a noise signal.

[0033] Step 102, extracting a sample frequency domain feature of the sample noisy speech signal.

[0034] In this embodiment, the execution subject described above can extract a sample frequency domain feature of the sample noisy speech signal. The sample frequency domain feature can include but is not limited to ERB (Equivalent Rectangular Bandwidth) cepstrum coefficient, first-order difference and second-order difference of ERB cepstrum coefficient, DCT (Discrete Cosine Transform) coefficient of speech pitch correlation degree, pitch period, etc.

[0035] Here, the sample noisy speech signal is usually a time domain signal, which needs to be converted to the frequency domain to extract the frequency domain feature. Specifically, the sample noisy speech signal is converted to the frequency domain to obtain a sample frequency domain noisy speech signal; the sample frequency domain feature is extracted from the sample frequency domain noisy speech signal.

[0036] Step 103, calculating a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal.

[0037] In this embodiment, the execution subject described above can calculate a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal. The sample target frequency band gain and the sample target time domain energy ratio can be calculated based on the sample noisy speech signal and its corresponding sample clean speech signal.

[0038] Step 104, training the network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision to obtain a noise reduction model.

[0039] In this embodiment, the execution subject described above can train the network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision to obtain a noise reduction model.

[0040] Generally, the sample frequency domain feature is input to the network, and the sample predicted frequency band gain and the sample predicted time domain energy ratio can be learned. Based on the difference between the sample predicted frequency band gain and the sample target frequency band gain, and the difference between the sample predicted time domain energy ratio and the sample target time domain energy ratio, the parameters of the network are adjusted, so that the above two differences are small enough, that is, the noise reduction model is obtained.

[0041] The noise reduction model training method provided by the embodiments of the present disclosure has good noise reduction performance and real-time performance. The noise reduction model has good suppression ability for stationary noise and non-stationary noise.

[0042] With reference to Figure 2 Fig. 200 shows another embodiment of the noise reduction model training method according to the present disclosure. The noise reduction model training method comprises the following steps:

[0043] Step 201, obtaining a sample clean speech signal and a sample noise signal.

[0044] In this embodiment, the execution subject of the noise reduction model training method can obtain a sample clean speech signal and a sample noise signal. The sample clean speech signal can be a speech signal collected in a quiet environment. The sample noise signal can be a noise signal collected in a noisy environment.

[0045] Step 202, fusing the sample clean speech signal and the sample noise signal to obtain a sample noisy speech signal.

[0046] In this embodiment, the execution subject can fuse the sample clean speech signal and the sample noise signal to obtain a sample noisy speech signal.

[0047] Here, the sample is generated by fusing the clean speech signal and the noise signal, so that the sample is easier to obtain.

[0048] Step 203, converting the sample noisy speech signal to the frequency domain to obtain a sample frequency domain noisy speech signal.

[0049] In this embodiment, the execution subject can convert the sample noisy speech signal to the frequency domain to obtain a sample frequency domain noisy speech signal.

[0050] Generally, the sample noisy speech signal can be a time domain signal. The sample noisy speech signal is first processed by Hamming windowing, and then Fourier transformed to obtain the sample frequency domain noisy speech signal.

[0051] Step 204, converting the sample frequency domain noisy speech signal to an equivalent rectangular bandwidth (ERB) domain to obtain a sample ERB domain frequency point.

[0052] In the embodiment, the execution subject can convert the sample frequency domain band noise speech signal into the ERB domain to obtain a sample ERB domain frequency point.

[0053] Generally, the sample frequency domain band noise speech signal can be converted into the ERB domain by using formula (1):

[0054] X erb (w) = 9.265 * loge (1+X(w) / (24.7*9.265)) (1)

[0055] wherein, X erb (w) is the sample ERB domain frequency point, and X(w) is the sample frequency domain band noise speech signal.

[0056] Step 205, dividing the sample ERB domain frequency point into M sample ERB domain subbands.

[0057] In the embodiment, the execution subject can divide the sample ERB domain frequency point into M sample ERB domain subbands. Wherein, M is a positive integer. For example, the sample ERB domain frequency point is evenly divided into 30 sample ERB domain subbands.

[0058] Step 206, converting the M sample ERB domain subbands into the frequency domain to obtain M sample frequency domain subbands.

[0059] In the embodiment, the execution subject can convert the M sample ERB domain subbands into the frequency domain to obtain M sample frequency domain subbands. Wherein, one sample ERB domain subband is converted to obtain one sample frequency domain subband.

[0060] Generally, the sample ERB domain subband can be converted into the frequency domain by using formula (2):

[0061]

[0062] wherein, X bin (w) is the sample ERB domain subband, 1≤bin≤M, and X erb (w) is the sample ERB domain subband.

[0063] Step 207, respectively performing discrete cosine transform (DCT) on the logarithmic energy sum of the M sample frequency domain subbands to obtain M sample DCT coefficients.

[0064] In the embodiment, the execution subject can respectively perform DCT on the logarithmic energy sum of the M sample frequency domain subbands to obtain M sample DCT coefficients.

[0065] Generally, the logarithmic energy sum of the sample frequency domain subband can be DCT by using formula (3):

[0066]

[0067] wherein y k is a sample DCT coefficient, C k is a coefficient, 0≤k≤M-1, E bin is a log energy sum of a sample frequency subband.

[0068] wherein C k may be defined by equation (4):

[0069]

[0070] Step 208, respectively calculating a first-order difference sum and a second-order difference sum of the first N sample DCT coefficients in the M sample DCT coefficients.

[0071] In the embodiment, the above execution subject can respectively calculate a first-order difference sum and a second-order difference sum of the first N sample DCT coefficients in the M sample DCT coefficients. Wherein, N is a positive integer, and N≤M. For example, respectively calculating a first-order difference sum and a second-order difference sum of the first 11 sample DCT coefficients in the 30 sample DCT coefficients.

[0072] Generally, a first-order difference sum of a sample DCT coefficient can be calculated by equation (5), and a second-order difference sum of a sample DCT coefficient can be calculated by equation (6):

[0073] y1 k = y l,k - y l-2,k (5)

[0074] y2 k = y l,k - 2*y l1,k + y l-2,k (6)

[0075] Wherein, l is a frame number, y1 k is a first-order difference sum of a sample DCT coefficient, equal to a kth sample DCT coefficient of a current frame minus a kth sample DCT coefficient of a previous two frames, y2 k is a second-order difference sum of a sample DCT coefficient, equal to a kth sample DCT coefficient of a current frame minus 2 times a kth sample DCT coefficient of a previous frame, plus a kth sample DCT coefficient of a previous two frames.

[0076] Step 209, calculating a P-dimensional DCT coefficient of a speech pitch correlation degree of a sample frequency band noise speech signal and a pitch period.

[0077] In this embodiment, the aforementioned execution entity can calculate the P-dimensional DCT coefficients and pitch period of the speech pitch correlation of the sample frequency-domain noisy speech signal. Here, P is a positive integer. For example, the 11-dimensional DCT coefficients and pitch period of the speech pitch correlation of the sample frequency-domain noisy speech signal can be calculated. It should be noted that the method for calculating the P-dimensional DCT coefficients and pitch period of the speech pitch correlation is prior art and will not be elaborated upon here.

[0078] Here, DCT coefficients, first-order and second-order differences of DCT coefficients, pitch correlation, and pitch period are extracted to enrich the feature content input into the network.

[0079] Step 210: Calculate the target frequency band gain of the samples based on the M sample frequency domain sub-bands of the clean speech signal and the M sample frequency domain sub-bands of the noisy speech signal.

[0080] In this embodiment, the aforementioned execution entity can calculate the target frequency band gain of the samples based on M sample frequency domain sub-bands of the clean speech signal and M sample frequency domain sub-bands of the noisy speech signal. For example, the target frequency band gain of 30 samples can be calculated based on 30 sample frequency domain sub-bands of the clean speech signal and 30 sample frequency domain sub-bands of the noisy speech signal.

[0081] Typically, the sample target bandwidth gain can be calculated using formula (7):

[0082] G bin =(∑ w X clean_bin (w)) / (∑ w X noisy_bin (w)) (7)

[0083] Among them, G bin X represents the target frequency band gain of the sample, and X is the ratio of the sample frequency domain subband of the clean speech signal to the sample frequency domain subband of the noisy speech signal. clean_bin (w) represents the sample frequency domain subband of the clean speech signal, X noisy_bin (w) represents the sample frequency domain subband of the sample noisy speech signal.

[0084] Step 211: Calculate the target time-domain energy ratio of the sample based on the clean speech signal and the noisy speech signal.

[0085] In this embodiment, the aforementioned execution entity can calculate the target time-domain energy ratio of a sample based on the clean speech signal and the noisy speech signal. For example, it can calculate the target time-domain energy ratio of one sample based on the clean speech signal and the noisy speech signal.

[0086] Typically, the sample target time-domain energy ratio can be calculated using formula (8):

[0087] G = (∑ t x clean (t)) / (∑ t x noisy (t)) (8)

[0088] Wherein, G is the sample target time domain energy ratio, is the energy ratio of the sample clean speech signal and the sample noisy speech signal, x clean (t) is the sample clean speech signal, x noisy (t) is the sample noisy speech signal.

[0089] Step 212, input the sample frequency domain features into the network, and learn to obtain the sample predicted frequency band gain and the sample predicted time domain energy ratio.

[0090] In the embodiment, the above execution subject can input the sample frequency domain features into the network, and the network learns two objects to obtain the sample predicted frequency band gain and the sample predicted time domain energy ratio. The sample predicted frequency band gain is M, and the value range is between 0-1, which is used for weighted gain of the frequency spectrum of the sample noisy speech signal. The number of the sample predicted time domain energy ratio is 1, which is used for assisting network training, reducing voice loss caused by noise reduction, and reflecting the component proportion or intensity of the sample clean speech signal in the sample noisy speech signal, and can be used for rough voice activity detection estimation.

[0091] Step 213, calculate the first loss function based on the sample target frequency band gain and the sample predicted frequency band gain.

[0092] In the embodiment, the above execution subject can calculate the first loss function based on the sample target frequency band gain and the sample predicted frequency band gain.

[0093] Generally, the first loss function can be calculated by formula (9):

[0094]

[0095] Wherein, is the first loss function, C0 and C1 are used to adjust the degree of loss, in practice, C0 = 10, C1 = 10, which can achieve better training effect, G bin is the sample target frequency band gain, is the sample predicted frequency band gain.

[0096] Step 214, calculate the second loss function based on the sample target time domain energy ratio and the sample predicted time domain energy ratio.

[0097] In the embodiment, the above execution subject can calculate the second loss function based on the sample target time domain energy ratio and the sample predicted time domain energy ratio.

[0098] Generally, the second loss function can be calculated by formula (10):

[0099]

[0100] wherein, is the second loss function, C2 is used to adjust the degree of loss, in practice, C2 = 5, which can achieve better training effect, G is the sample target domain energy ratio, is the sample prediction domain energy ratio.

[0101] Step 215, adjusting the parameters of the network based on the first loss function and the second loss function to obtain the denoising model.

[0102] In the embodiment, the above execution subject can adjust the parameters of the network based on the first loss function and the second loss function to obtain the denoising model.

[0103] Generally, the first loss function and the second loss function are weighted and summed to obtain a total loss function, and the parameters of the network are adjusted based on the total loss function to reduce the value of the total loss function until the network converges, that is, the denoising model can be obtained.

[0104] As can be seen from Figure 2 , compared with Figure 1 the corresponding embodiment, the flow 200 of the denoising model training method in the embodiment highlights the sample acquisition step, the frequency domain feature extraction step and the loss calculation step. Therefore, the scheme described in the embodiment generates samples by fusing clean speech signals and noise signals, so that the acquisition of samples is more convenient. Extracting DCT coefficients, first-order differences and second-order differences of DCT coefficients, pitch correlation and pitch period makes the feature content input into the network more rich. Calculating the loss function for frequency band gain training and the loss function for time domain energy ratio gain training together as the training target of the network, so as to train a better denoising model.

[0105] In order to facilitate understanding, Figure 3 a schematic diagram of training features in the denoising model training method in Figure 2 is shown. As Figure 3 shown, the sample clean speech signal is fused with the sample noise signal to generate a sample noisy speech signal. Based on the sample noisy speech signal, ERB frequency band division and DCT are performed to obtain 30-dimensional DCT coefficients, 11-dimensional first-order differences and 11-dimensional second-order differences. Based on the sample noisy speech signal, pitch extraction is performed to obtain 11-dimensional pitch correlation and pitch period. Based on the sample clean speech signal and the sample noisy speech signal, the frequency domain subband energy ratio and the time domain energy ratio are calculated to obtain 1 sample time domain energy ratio and 30 sample frequency band gains as the training target.

[0106] Furthermore, Figure 4 It shows Figure 2 The network structure diagram of the denoising model in the training method of the denoising model. (Example) Figure 4 As shown, 64 features, including 30-dimensional DCT coefficients, 11-dimensional first-order and second-order differences, 11-dimensional pitch correlation, and pitch period, are used as inputs to the network. The network consists of one fully connected layer Dense (96), three gated recurrent units (GRUs) (96), one fully connected layer Dense (30), and one fully connected layer Dense (1). The fully connected layer Dense (30) outputs 30 frequency band gains G. bin The energy ratio of G to 1 time domain energy.

[0107] Figure 5 A flow 500 of an embodiment of a speech denoising method according to the present disclosure is shown. The speech denoising method includes the following steps:

[0108] Step 501: Obtain the noisy speech signal.

[0109] In this embodiment, the entity executing the speech denoising method can acquire a noisy speech signal. The noisy speech signal is a speech signal with noise, which can be a speech signal collected in a noisy environment.

[0110] Step 502: Extract the frequency domain features of the noisy speech signal.

[0111] In this embodiment, the aforementioned execution entity can extract the frequency domain features of the noisy speech signal. These frequency domain features may include, but are not limited to: ERB cepstral coefficients, the first and second differences of the ERB cepstral coefficients, and the DCT coefficients and pitch period of the speech pitch correlation, etc.

[0112] Here, the noisy speech signal is usually a time-domain signal, which needs to be converted to the frequency domain in order to extract frequency domain features. Specifically, the noisy speech signal is converted to the frequency domain to obtain a frequency-domain noisy speech signal; frequency domain features are then extracted from the frequency-domain noisy speech signal.

[0113] It should be noted that the frequency domain feature extraction method can be referenced. Figure 2 The frequency domain feature extraction method for the samples is not described in detail here.

[0114] Step 503: Input the frequency domain features into the noise reduction model to obtain M frequency band gains.

[0115] In this embodiment, the aforementioned execution entity can input frequency domain features into the noise reduction model to obtain M frequency band gains. Here, M is a positive integer, and the noise reduction model can be... Figure 1 or Figure 2The denoising model training method shown is trained, and details are not repeated here.

[0116] Generally, the frequency domain features are input into the denoising model, M frequency band gains and 1 time domain energy ratio can be obtained, and the denoising only uses the M frequency band gains.

[0117] It should be noted that the trained denoising model can be converted into a TensorflowLite model first, and the denoising process is called by TensorflowLite.

[0118] Step 504, denoising based on M frequency band gains to obtain a clean speech signal.

[0119] In this embodiment, the execution subject can perform denoising based on the M frequency band gains to obtain a clean speech signal.

[0120] Generally, based on the M frequency band gains, a frequency domain clean speech signal can be obtained, and the frequency domain clean speech signal is converted to the time domain to obtain a clean speech signal.

[0121] The voice denoising method provided in the embodiments of the present disclosure has good suppression ability for both stationary noise and non-stationary noise. It can be applied to devices with audio and video functions, such as sound boxes with screens, mobile phones, computers and other devices, to suppress noise, improve sound quality and improve user call experience.

[0122] Continuing to refer to Figure 6 which shows a flow 600 of still another embodiment of the voice denoising method according to the present disclosure. The voice denoising method includes the following steps:

[0123] Step 601, obtaining a noisy speech signal.

[0124] In this embodiment, the specific operation of step 601 has been described in detail in Figure 5 The step 501 in the embodiment shown is described in detail, and details are not repeated here.

[0125] Step 602, converting the noisy speech signal to the frequency domain to obtain a frequency domain noisy speech signal.

[0126] In this embodiment, the execution subject of the voice denoising method can convert the noisy speech signal to the frequency domain to obtain a frequency domain noisy speech signal.

[0127] Generally, the noisy speech signal can be a time domain signal. The noisy speech signal is first processed by Hamming windowing, and then Fourier transformed, so that the frequency domain noisy speech signal can be obtained.

[0128] Step 603, extracting frequency domain features from the frequency domain noisy speech signal.

[0129] In the embodiment, the execution subject can extract the frequency domain features from the frequency domain band noise speech signal. The frequency domain features can include, but are not limited to, ERB cepstrum coefficients, first-order and second-order differences of the ERB cepstrum coefficients, DCT coefficients of speech pitch correlation, pitch period, and the like.

[0130] It should be noted that the frequency domain feature extraction method can refer to the sample frequency domain feature extraction method in Figure 2 , which will not be described here.

[0131] In step 604, the frequency domain features are input into the noise reduction model to obtain M frequency band gains.

[0132] In the embodiment, the specific operation of step 604 has been described in detail in step 503 of the embodiment shown in Figure 5 , which will not be described here.

[0133] In step 605, the M frequency band gains are interpolated, and the frequency points of the interpolated frequency band gains are weighted to obtain a frequency domain clean speech signal.

[0134] In the embodiment, the execution subject can interpolate the M frequency band gains, and weight the frequency points of the interpolated frequency band gains, so as to obtain the frequency domain clean speech signal.

[0135] In step 606, the frequency domain clean speech signal is converted to the time domain to obtain a clean speech signal.

[0136] In the embodiment, the execution subject can convert the frequency domain clean speech signal to the time domain to obtain the clean speech signal.

[0137] Generally, the frequency domain clean speech signal is inversely Fourier transformed to obtain the clean speech signal.

[0138] In order to facilitate understanding, Figure 7 shows a flowchart of the speech noise reduction method in Figure 6 . As shown in Figure 7 , the noisy speech signal is Fourier transformed to obtain a frequency domain band noise speech signal. The frequency domain band noise speech signal is feature extracted to obtain frequency domain features. The frequency domain features are input into the Tflite model converted from the noise reduction model to obtain frequency band gains. The frequency band gains are gain interpolated to obtain a frequency domain clean speech signal. The frequency domain clean speech signal is inversely Fourier transformed to obtain a clean speech signal. Figure 8 and Figure 9 respectively show the noise reduction effect schematic diagram of the speech noise reduction method of the embodiment. The upper one is a noisy audio, and the lower one is a noise reduction audio. It can be seen that the speech noise reduction method of the embodiment can well suppress noise and improve sound quality.

[0139] For ease of understanding, Figure 10 A scene diagram illustrating a method for training a noise reduction model and a method for speech noise reduction is shown. As Figure 10 shown, feature extraction is performed on the training data, and the obtained features are used for model training to obtain a noise reduction model. Feature extraction is performed on the noisy speech, and the obtained features are input into the noise reduction model for noise reduction to obtain clean speech.

[0140] Further reference Figure 11 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a noise reduction model training device, which corresponds to the method embodiment shown in Figure 1 , and the device can be specifically applied to various electronic devices.

[0141] As Figure 11 shown, the noise reduction model training device 1100 of the present embodiment can include an acquisition module 1101, an extraction module 1102, a calculation module 1103, and a training module 1104. The acquisition module 1101 is configured to acquire a sample noisy speech signal. The extraction module 1102 is configured to extract a sample frequency domain feature of the sample noisy speech signal. The calculation module 1103 is configured to calculate a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal. The training module 1104 is configured to train a network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision, to obtain a noise reduction model.

[0142] In the present embodiment, in the noise reduction model training device 1100: the specific processing of the acquisition module 1101, the extraction module 1102, the calculation module 1103, and the training module 1104 and the technical effects brought by the same can be respectively referred to the related descriptions of steps 101-104 in the corresponding embodiments, which will not be repeated here. Figure 1

[0143] In some optional implementations of the present embodiment, the extraction module 1102 includes a conversion submodule configured to convert the sample noisy speech signal to a frequency domain to obtain a sample frequency domain noisy speech signal, and an extraction submodule configured to extract the sample frequency domain feature from the sample frequency domain noisy speech signal.

[0144] ​In some optional implementation forms of the embodiment, the extraction sub-module comprises: a first conversion unit configured to convert the sample frequency-domain band noisy speech signal into an equivalent rectangular bandwidth (ERB) domain to obtain sample ERB domain frequency points; a division unit configured to divide the sample ERB domain frequency points into M sample ERB domain subbands, where M is a positive integer; and a second conversion unit configured to convert the M sample ERB domain subbands into the frequency domain to obtain M sample frequency-domain subbands; and a transformation unit configured to perform discrete cosine transform (DCT) on the logarithmic energies of the M sample frequency-domain subbands, respectively, to obtain M sample DCT coefficients.

[0145] In some optional implementation forms of the embodiment, the extraction sub-module further comprises a first calculation unit configured to calculate first-order differences and second-order differences of the first N sample DCT coefficients among the M sample DCT coefficients, respectively, where N is a positive integer and N≤M.

[0146] In some optional implementation forms of the embodiment, the extraction sub-module further comprises a second calculation unit configured to calculate P-dimensional DCT coefficients of the pitch correlation degree of the sample frequency-domain band noisy speech signal and the pitch period, where P is a positive integer.

[0147] In some optional implementation forms of the embodiment, the acquisition module 1101 is further configured to: acquire a sample clean speech signal and a sample noise signal; and fuse the sample clean speech signal and the sample noise signal to obtain a sample band noisy speech signal; and the calculation module 1103 is further configured to: calculate a sample target frequency band gain based on the M sample frequency-domain subbands of the sample clean speech signal and the M sample frequency-domain subbands of the sample band noisy speech signal; and calculate a sample target time domain energy ratio based on the sample clean speech signal and the sample band noisy speech signal.

[0148] In some optional implementation forms of the embodiment, the training module 1104 is further configured to: input the sample frequency-domain features into the network to learn a sample predicted frequency band gain and a sample predicted time domain energy ratio; calculate a first loss function based on the sample target frequency band gain and the sample predicted frequency band gain; calculate a second loss function based on the sample target time domain energy ratio and the sample predicted time domain energy ratio; and adjust parameters of the network based on the first loss function and the second loss function to obtain the noise reduction model.

[0149] Further reference Figure 12 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a speech noise reduction device, which corresponds to the method embodiment shown in Figure 5 , and the device can be applied in various electronic devices.

[0150] As Figure 12As shown, the speech noise reduction device 1200 in this embodiment can include an obtaining module 1201, an extracting module 1202, an inputting module 1203, and a noise reduction module 1204. The obtaining module 1201 is configured to obtain a noisy speech signal. The extracting module 1202 is configured to extract a frequency domain feature of the noisy speech signal. The inputting module 1203 is configured to input the frequency domain feature to a noise reduction model to obtain M frequency band gains, where M is a positive integer, and the noise reduction model is obtained by training the device as shown. Figure 5 or Figure 6 The noise reduction module 1204 is configured to perform noise reduction based on the M frequency band gains to obtain a clean speech signal.

[0151] In this embodiment, in the speech noise reduction device 1200: the specific processing of the obtaining module 1201, the extracting module 1202, the inputting module 1203, and the noise reduction module 1204 and the technical effects brought by the specific processing can be respectively referred to the related descriptions of steps 501-504 in the corresponding embodiments, which will not be repeated here. Figure 5

[0152] In some optional implementations of this embodiment, the extracting module 1202 is further configured to: convert the noisy speech signal to a frequency domain to obtain a frequency domain noisy speech signal; and extract the frequency domain feature from the frequency domain noisy speech signal. The noise reduction module 1204 is further configured to: interpolate the M frequency band gains, and weight the frequency points of the interpolated frequency band gains to obtain a frequency domain clean speech signal; and convert the frequency domain clean speech signal to a time domain to obtain the clean speech signal.

[0153] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0154] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0155] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document. ​

[0156] like Figure 13 As shown, device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1302 or a computer program loaded from storage unit 1308 into random access memory (RAM) 1303. The RAM 1303 may also store various programs and data required for the operation of device 1300. The computing unit 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0157] Multiple components in device 1300 are connected to I / O interface 1305, including: input unit 1306, such as keyboard, mouse, etc.; output unit 1307, such as various types of monitors, speakers, etc.; storage unit 1308, such as disk, optical disk, etc.; and communication unit 1309, such as network card, modem, wireless transceiver, etc. Communication unit 1309 allows device 1300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0158] The computing unit 1301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as denoising model training methods or speech denoising methods. For example, in some embodiments, the denoising model training method or speech denoising method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1300 via ROM 1302 and / or communication unit 1309. When the computer program is loaded into RAM 1303 and executed by the computing unit 1301, one or more steps of the denoising model training method or speech denoising method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured by any other suitable means (e.g., by means of firmware) to perform a noise reduction model training method or a speech noise reduction method.

[0159] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0160] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0161] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0162] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0163] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0164] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0165] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without departing from the desired results provided by the technical solutions of the present disclosure, and this is not limited herein.

[0166] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalent replacements, improvements, and the like made within the spirit and principles of the present disclosure should be included within the scope of the present disclosure.

Claims

1. A method for training a noise reduction model, comprising: obtaining a sample noisy speech signal; extracting a sample frequency domain feature of the sample noisy speech signal; calculating a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal; training a network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision to obtain a noise reduction model; the extracting a sample frequency domain feature of the sample noisy speech signal comprises: converting the sample noisy speech signal to a frequency domain to obtain a sample frequency domain noisy speech signal; extracting the sample frequency domain feature from the sample frequency domain noisy speech signal; the extracting the sample frequency domain feature from the sample frequency domain noisy speech signal comprises: converting the sample frequency domain noisy speech signal to an equivalent rectangular bandwidth (ERB) domain to obtain a sample ERB domain frequency point; dividing the sample ERB domain frequency point into M sample ERB domain subbands, where M is a positive integer; converting the M sample ERB domain subbands to a frequency domain to obtain M sample frequency domain subbands; respectively performing a discrete cosine transform (DCT) on the logarithmic energy of the M sample frequency domain subbands to obtain M sample DCT coefficients.

2. The method of claim 1, wherein, the extracting the sample frequency domain feature from the sample frequency domain noisy speech signal further comprises: respectively calculating a first-order difference and a second-order difference of the first N sample DCT coefficients in the M sample DCT coefficients, where N is a positive integer and N≤M.

3. The method of claim 1 or 2, wherein, the extracting the sample frequency domain feature from the sample frequency domain noisy speech signal further comprises: calculating a P-dimensional DCT coefficient of a speech pitch correlation degree of the sample frequency domain noisy speech signal and a pitch period, where P is a positive integer.

4. The method of claim 1, wherein, the obtaining a sample noisy speech signal comprises: obtaining a sample clean speech signal and a sample noise signal; fusing the sample clean speech signal and the sample noise signal to obtain the sample noisy speech signal; and the calculating a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal comprises: calculating the sample target frequency band gain based on M sample frequency domain subbands of the sample clean speech signal and M sample frequency domain subbands of the sample noisy speech signal; calculating the sample target time domain energy ratio based on the sample clean speech signal and the sample noisy speech signal.

5. The method of any one of claims 1-4, wherein, the training a network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision to obtain a noise reduction model comprises: inputting the sample frequency domain feature into the network to learn a sample predicted frequency band gain and a sample predicted time domain energy ratio; calculating a first loss function based on the sample target frequency band gain and the sample predicted frequency band gain; calculating a second loss function based on the sample target time domain energy ratio and the sample predicted time domain energy ratio; adjusting parameters of the network based on the first loss function and the second loss function to obtain the noise reduction model.

6. A method for reducing noise of speech, comprising: obtaining a noisy speech signal; extracting a frequency domain feature of the noisy speech signal; inputting the frequency domain feature of the noisy speech signal into a noise reduction model to obtain M frequency band gains, wherein M is a positive integer, and the noise reduction model is trained by using the method in any one of claims 1-5; performing noise reduction based on the M frequency band gains to obtain a clean speech signal.

7. The method of claim 6, wherein, The extracting the frequency domain feature of the noisy speech signal comprises: converting the noisy speech signal into a frequency domain to obtain a frequency domain noisy speech signal; extracting the frequency domain feature from the frequency domain noisy speech signal; and The performing noise reduction based on the M frequency band gains to obtain a clean speech signal comprises: interpolating the M frequency band gains, and weighting frequency points of the interpolated frequency band gains to obtain a frequency domain clean speech signal; converting the frequency domain clean speech signal into a time domain to obtain the clean speech signal.

8. A noise reduction model training apparatus, comprising: an obtaining module configured to obtain a sample noisy speech signal; an extracting module configured to extract a sample frequency domain feature of the sample noisy speech signal; a calculating module configured to calculate a sample target frequency band gain and a sample target time domain energy ratio of the sample noisy speech signal; a training module configured to train a network by taking the sample frequency domain feature as input and taking the sample target frequency band gain and the sample target time domain energy ratio as supervision to obtain a noise reduction model; The extracting module comprises: a converting submodule configured to convert the sample noisy speech signal into a frequency domain to obtain a sample frequency domain noisy speech signal; an extracting submodule configured to extract the sample frequency domain feature from the sample frequency domain noisy speech signal; The extracting submodule comprises: a first converting unit configured to convert the sample frequency domain noisy speech signal into an equivalent rectangular bandwidth (ERB) domain to obtain sample ERB domain frequency points; a dividing unit configured to divide the sample ERB domain frequency points into M sample ERB domain subbands, wherein M is a positive integer; a second converting unit configured to convert the M sample ERB domain subbands into a frequency domain to obtain M sample frequency domain subbands; a transforming unit configured to respectively perform discrete cosine transform (DCT) on log energies of the M sample frequency domain subbands to obtain M sample DCT coefficients.

9. The apparatus of claim 8, wherein, The extracting submodule further comprises: a first calculating unit configured to respectively calculate first-order differences and second-order differences of first N sample DCT coefficients in the M sample DCT coefficients, wherein N is a positive integer and N≤M.

10. The apparatus of claim 8 or 9, wherein, The extracting submodule further comprises: a second calculating unit configured to calculate P-dimensional DCT coefficients of a speech pitch correlation degree of the sample frequency domain noisy speech signal and a pitch period, wherein P is a positive integer.

11. The apparatus of claim 8, wherein, The obtaining module is further configured to: obtain a sample clean speech signal and a sample noise signal; fuse the sample clean speech signal and the sample noise signal to obtain the sample noisy speech signal; and The calculating module is further configured to: calculate the sample target frequency band gain based on M sample frequency domain subbands of the sample clean speech signal and M sample frequency domain subbands of the sample noisy speech signal; Based on the sample clean speech signal and the sample noisy speech signal, a sample target time domain energy ratio is calculated.

12. The apparatus of any one of claims 8-11, wherein, The training module is further configured to: input the sample frequency domain features into the network to learn a sample predicted frequency band gain and a sample predicted time domain energy ratio; based on the sample target frequency band gain and the sample predicted frequency band gain, a first loss function is calculated; based on the sample target time domain energy ratio and the sample predicted time domain energy ratio, a second loss function is calculated; based on the first loss function and the second loss function, the parameters of the network are adjusted to obtain the noise reduction model.

13. A speech noise reduction device, comprising: an acquisition module configured to acquire a noisy speech signal; an extraction module configured to extract frequency domain features of the noisy speech signal; an input module configured to input the frequency domain features into a noise reduction model to obtain M frequency band gains, wherein M is a positive integer, and the noise reduction model is trained by the device of any one of claims 8-12; a noise reduction module configured to perform noise reduction based on the M frequency band gains to obtain a clean speech signal.

14. The apparatus of claim 13, wherein, The extraction module is further configured to: convert the noisy speech signal to the frequency domain to obtain a frequency domain noisy speech signal; extract the frequency domain features from the frequency domain noisy speech signal; and The noise reduction module is further configured to: interpolate the M frequency band gains, and weight the frequency points of the interpolated frequency band gains to obtain a frequency domain clean speech signal; convert the frequency domain clean speech signal to the time domain to obtain the clean speech signal.

15. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5 or the method of any one of claims 6-7.

16. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-5 or the method of any one of claims 6-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-5 or the method of any one of claims 6-7.

Images