Training method of noise reduction model, speech noise reduction method, device and electronic equipment

Abstract

The present disclosure provides a method for training a noise reduction model, a speech noise reduction method, an apparatus and an electronic device. The method comprises: obtaining source noisy speech data, and performing reverberation processing on the source noisy speech data to generate corresponding training samples, wherein the training samples comprise a plurality of samples, and each sample comprises multi-channel reverberation noisy speech data; training a to-be-trained speech noise reduction model based on the multi-channel reverberation noisy speech data included in the training samples to obtain a trained first speech noise reduction model. In the present disclosure, the human ear has a good listening experience for the noise reduction speech output by the trained speech noise reduction model, improves the propagation accuracy of the information carried in the speech data, reduces the influence of noise on the propagation of the information carried in the speech data, improves the robustness of the speech noise reduction model, strengthens the applicability and practicality of the speech noise reduction model, and optimizes the training method and training effect of the speech noise reduction model.

Description

Technical Field

[0001] The present disclosure relates to the field of communications, and in particular to a noise reduction model training method, a speech noise reduction method, a device, and an electronic device. Background Art

[0002] With the development of technology, people are increasingly relying on mobile devices for video or audio calls. During a call, they may encounter noisy background noise or sudden noise. Therefore, it is necessary to perform noise reduction processing on the related noise during the call to ensure the clarity of people's voice during the call.

[0003] In the related art, the method for performing noise reduction processing on noisy speech has the problem of being slow, unable to guarantee the efficiency of the noise reduction processing, and having poor real-time performance. Summary of the Invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the first aspect of the present disclosure proposes a training method for a noise reduction model.

[0006] A second aspect of the present disclosure provides a method for reducing speech noise.

[0007] A third aspect of the present disclosure provides a training device for a noise reduction model.

[0008] A fourth aspect of the present disclosure provides a speech noise reduction device.

[0009] A fifth aspect of the present disclosure provides an electronic device.

[0010] A sixth aspect of the present disclosure provides a computer-readable storage medium.

[0011] In a first aspect, the present disclosure proposes a method for training a noise reduction model, comprising: obtaining source noisy speech data, and performing reverberation processing on the source noisy speech data to generate corresponding training samples, wherein the training samples include multiple samples, each sample including multi-channel reverberated noisy speech data; and training a speech noise reduction model to be trained based on the multi-channel reverberated noisy speech data included in the training samples to obtain a trained first speech noise reduction model.

[0012] In addition, the training method of the denoising model proposed in the first aspect of the present disclosure may also have the following additional technical features:

[0013] According to one embodiment of the present disclosure, a speech noise reduction model to be trained is trained based on the multi-channel reverberant and noisy speech data included in the training samples to obtain a trained first speech noise reduction model, including: performing short-time Fourier transform on the multi-channel reverberant and noisy speech data included in the training samples frame by frame to generate a short-time spectrum of a single frame of multi-channel reverberant speech data; and training the speech noise reduction model to be trained based on the short-time spectrum to obtain a trained first speech noise reduction model.

[0014] According to one embodiment of the present disclosure, a speech noise reduction model to be trained is trained based on a short-time spectrum to obtain a trained first speech noise reduction model, including: performing time-frequency masking on the short-time spectrum to obtain a spectrum mask of the short-time spectrum; obtaining a denoised short-time spectrum after denoising the short-time spectrum based on the spectrum mask and the short-time spectrum; performing a short-time inverse Fourier transform on the denoised short-time spectrum to generate speech data of the denoised short-time spectrum as single-frame denoised speech data as a training output of the speech noise reduction model; obtaining a training loss function of the speech noise reduction model based on the single-frame denoised speech data, and adjusting parameters of the speech noise reduction model based on the training loss function until the training is completed to obtain the trained first speech noise reduction model.

[0015] According to one embodiment of the present disclosure, time-frequency masking is performed on a short-time spectrum to obtain a spectrum mask of the short-time spectrum, including: extracting single-frame noisy speech features of the short-time spectrum through a feature extraction layer of a speech denoising model; obtaining masking parameters of the single-frame noisy speech features, and performing time-frequency masking on the single-frame noisy speech features according to the masking parameters to obtain a spectrum mask.

[0016] According to one embodiment of the present disclosure, a single-frame noisy speech feature of a short-time spectrum is extracted through a speech noise reduction model, including: extracting noisy speech features in the frequency domain and time domain of the short-time spectrum through a feature extraction layer to obtain a single-frame noisy speech feature of the short-time spectrum.

[0017] According to one embodiment of the present disclosure, a feature extraction layer is used to extract noisy speech features in the frequency domain and time domain of a short-time frequency to obtain single-frame noisy speech features of a short-time spectrum, including: extracting frequency domain features of the short-time spectrum based on a frequency domain causal hole convolution layer in the feature extraction layer to obtain frequency domain noisy speech features; extracting time domain features of the short-time spectrum based on a time domain causal hole convolution layer in the feature extraction layer to obtain time domain noisy speech features; and obtaining single-frame noisy speech features of a short-time spectrum based on the frequency domain noisy speech features and the time domain noisy speech features.

[0018] According to one embodiment of the present disclosure, any one of the frequency-domain noisy speech features and the time-domain noisy speech features includes: extracting the initial noisy speech features of the short-time spectrum in the causal hole convolution layer of the feature extraction layer; and obtaining any one of the noisy speech features of the short-time spectrum based on the initial noisy speech features and the sigmoid activation function.

[0019] According to one embodiment of the present disclosure, in a causal convolution layer of a feature extraction layer, initial noisy speech features of a short-time spectrum are extracted, including: performing a first feature extraction on the short-time spectrum input to the causal convolution layer, extracting a first frame feature of a speech frame corresponding to the short-time spectrum, and a noisy speech feature corresponding to the first frame feature; obtaining a first noisy speech feature of the short-time spectrum based on the first frame feature and the noisy speech feature corresponding to the first frame feature; partially discarding the first noisy speech feature, calculating the remaining features in the first noisy speech feature after discarding according to an ELU activation function, and using the calculated noisy speech feature as the initial noisy speech feature.

[0020] According to one embodiment of the present disclosure, any noisy speech feature of a short-time spectrum extracted by a causal convolution layer is obtained based on initial frequency domain features and a sigmoid activation function, including: performing a second feature extraction on the initial noisy speech feature to extract a second frame feature of a speech frame corresponding to the initial noisy speech feature and a noisy speech feature corresponding to the second frame feature; obtaining a second noisy speech feature of the short-time spectrum based on the second frame feature and the noisy speech feature corresponding to the second frame feature; performing a reference calculation on the second noisy speech feature based on the sigmoid activation function, and using the calculated noisy speech feature as a third noisy speech feature; normalizing the second noisy speech feature and the third noisy speech feature, and using the normalized noisy speech feature as any noisy speech feature of the short-time spectrum output by the causal convolution layer.

[0021] According to one embodiment of the present disclosure, when the causal convolution layer outputs frequency domain noisy speech features, the causal void convolution layer is a frequency domain causal void convolution layer; when the causal convolution layer outputs frequency domain noisy speech features, the causal void convolution layer is a time domain causal void convolution layer.

[0022] According to one embodiment of the present disclosure, a training loss function of a speech denoising model is obtained based on single-frame denoised speech data, and parameters of the speech denoising model are adjusted according to the training loss function until the training is completed to obtain a trained first speech denoising model, including: obtaining single-frame speech data in single-frame multi-channel reverberation-noised speech data, and obtaining a scale-invariant signal-to-noise ratio loss and a short-time objective intelligibility loss between the single-frame speech data and the single-frame denoised speech data; performing a weighted summation of the scale-invariant signal-to-noise ratio loss and the short-time objective intelligibility loss to obtain the training loss function of the speech denoising model.

[0023] According to an embodiment of the present disclosure, after obtaining the single-frame noisy speech feature of the short-time spectrum, the method includes: caching the single-frame noisy speech feature as a historical single-frame noisy speech feature in a cache unit to wait for the next call.

[0024] According to one embodiment of the present disclosure, the method further includes: performing distillation processing on the first speech denoising model to obtain a lightweight second speech denoising model.

[0025] According to one embodiment of the present disclosure, a first speech noise reduction model is distilled to obtain a lightweight second speech noise reduction model, including: obtaining a lightweight speech noise reduction model to be trained based on the first speech noise reduction model; obtaining the distillation loss of the lightweight speech noise reduction model based on the first speech noise reduction model based on the multi-channel reverberation and noisy speech data included in the training sample; adjusting the parameters of the lightweight speech noise reduction model according to the distillation loss until the training is completed, and obtaining a trained second speech noise reduction model.

[0026] According to one embodiment of the present disclosure, a lightweight speech noise reduction model to be trained is obtained based on the first speech noise reduction model, including: obtaining lightweight extraction conditions of the model structure of the first speech noise reduction model; constructing a model to be trained whose model structure meets the lightweight extraction conditions as the lightweight speech noise reduction model.

[0027] According to one embodiment of the present disclosure, based on the multi-channel reverberant and noisy speech data included in the training sample, the distillation loss of the lightweight speech denoising model based on the first speech denoising model is obtained, including: based on the multi-channel reverberant and noisy speech data included in the training sample, obtaining the feature extraction loss and noise reduction loss of the lightweight speech denoising model based on the first speech denoising model; performing weighted summation of the feature extraction loss and the noise reduction loss to obtain the distillation loss of the lightweight speech denoising model based on the first speech denoising model.

[0028] According to one embodiment of the present disclosure, a process for obtaining feature extraction loss includes: obtaining a first noisy speech feature of multi-channel reverberant and noisy speech data included in a training sample extracted by a first speech denoising model; obtaining a second noisy speech feature of the multi-channel reverberant and noisy speech data included in a training sample extracted by a lightweight speech denoising model; and obtaining a feature extraction loss of the lightweight speech denoising model based on the first speech denoising model according to the first noisy speech feature and the second noisy speech feature.

[0029] According to one embodiment of the present disclosure, the process of obtaining noise reduction loss includes: inputting the multi-channel reverberant and noisy speech data included in the training sample into the first speech noise reduction model to obtain the first noise-reduced speech output by the first speech noise reduction model; inputting the multi-channel reverberant and noisy speech data included in the training sample into the lightweight speech noise reduction model to obtain the second noise-reduced speech output by the lightweight speech noise reduction model; and obtaining the noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model according to the first noise-reduced speech and the second noise-reduced speech.

[0030] According to one embodiment of the present disclosure, source noisy speech data is obtained, and reverberation processing is performed on the source noisy speech data to generate corresponding training samples, wherein the training samples include multiple samples, each sample including multi-channel reverberation-noisy speech data, including: obtaining source speech data and source noise data in the source noisy speech data; performing reverberation processing on the source speech data to obtain multi-channel reverberation speech data of the source speech data; performing reverberation processing on the source noise data to obtain multi-channel reverberation noise data of the source noise data; mixing the multi-channel reverberation speech data and the multi-channel reverberation noise data to obtain multi-channel reverberation-noisy speech data, and generating training samples based on the multi-channel reverberation-noisy speech data.

[0031] According to one embodiment of the present disclosure, before obtaining source speech data and source noise data in source noisy speech data, the method includes: obtaining initial speech data and initial noise data; performing data cleaning and data enhancement on the initial speech data and the initial noise data, respectively, to obtain source speech data corresponding to the initial speech data and source noise data corresponding to the initial noise data; and splicing the source speech data and the source noise data to generate source noisy speech data.

[0032] According to one embodiment of the present disclosure, multi-channel reverberation speech data and multi-channel reverberation noise data are mixed to obtain multi-channel reverberation-noisy speech data, including: adjusting the speech length of the multi-channel reverberation noise data to be the same as the speech length of the multi-channel reverberation speech data; mixing the multi-channel reverberation noise data and the multi-channel reverberation speech data with the same speech length to obtain the mixed multi-channel reverberation-noisy speech data, and generating training samples based on the multi-channel reverberation-noisy speech data.

[0033] According to one embodiment of the present disclosure, the method further includes: obtaining evaluation reverberant and noisy speech data for evaluation based on the multi-channel reverberant and noisy speech data; and evaluating the first speech noise reduction model and / or the second speech noise reduction model based on the evaluation reverberant and noisy speech data.

[0034] According to one embodiment of the present disclosure, the first speech noise reduction model and / or the second speech noise reduction model are evaluated based on the evaluation of reverberant and noisy speech data, including: obtaining evaluation parameters of the first speech noise reduction model and / or the second speech noise reduction model, and evaluating the first speech noise reduction model and / or the second speech noise reduction model based on the evaluation parameters.

[0035] To achieve the above-mentioned purpose, the second aspect of the present disclosure proposes a speech denoising method, which includes: obtaining a multi-channel reverberant and noisy speech to be denoised, and performing short-time Fourier transform on the multi-channel reverberant and noisy speech data to be denoised frame by frame to generate a short-time spectrum of a single frame of the multi-channel reverberant and noisy speech data to be denoised; based on a trained target speech denoising model, performing denoising processing on the short-time spectrum to be denoised frame by frame to obtain the denoised target speech data of the multi-channel reverberant and noisy speech data to be denoised output by the target speech denoising model; wherein the target speech denoising model is trained by the denoising model training method of the first aspect mentioned above.

[0036] To achieve the above-mentioned purpose, the third aspect of the present disclosure proposes a training device for a noise reduction model, which includes: a reverberation module, which is used to obtain source noisy speech data and perform reverberation processing on the source noisy speech data to generate corresponding training samples, wherein the training samples include multiple samples, each sample includes multi-channel reverberated noisy speech data; a training module, which is used to train the speech noise reduction model to be trained based on the multi-channel reverberated noisy speech data included in the training samples to obtain a trained first speech noise reduction model.

[0037] In addition, the noise reduction model training device proposed in the third aspect of the present disclosure may also have the following additional technical features:

[0038] According to one embodiment of the present disclosure, the training module is further used to: perform short-time Fourier transform on the multi-channel reverberation and noisy speech data included in the training sample frame by frame to generate a short-time spectrum of the single-frame multi-channel reverberation speech data; and train the speech noise reduction model to be trained based on the short-time spectrum to obtain a trained first speech noise reduction model.

[0039] According to one embodiment of the present disclosure, the training module is further used to: perform time-frequency masking on the short-time spectrum to obtain a spectrum mask of the short-time spectrum; obtain a denoised short-time spectrum after denoising the short-time spectrum based on the spectrum mask and the short-time spectrum; perform a short-time inverse Fourier transform on the denoised short-time spectrum to generate speech data of the denoised short-time spectrum as single-frame denoised speech data as a training output of the speech denoising model; obtain a training loss function of the speech denoising model based on the single-frame denoised speech data, and adjust the parameters of the speech denoising model according to the training loss function until the training is completed, thereby obtaining a trained first speech denoising model.

[0040] According to one embodiment of the present disclosure, the training module is further used to: extract the single-frame noisy speech features of the short-time spectrum through the feature extraction layer of the speech noise reduction model; obtain the masking parameters of the single-frame noisy speech features, and perform time-frequency masking on the single-frame noisy speech features according to the masking parameters to obtain a spectrum mask.

[0041] According to one embodiment of the present disclosure, the training module is further used to: extract noisy speech features in the frequency domain and time domain of the short-time spectrum through the feature extraction layer to obtain single-frame noisy speech features of the short-time spectrum.

[0042] According to one embodiment of the present disclosure, the training module is further used to: perform frequency domain feature extraction on the short-time spectrum based on the frequency domain causal hole convolution layer in the feature extraction layer to obtain frequency domain noisy speech features; perform time domain feature extraction on the short-time spectrum based on the time domain causal hole convolution layer in the feature extraction layer to obtain time domain noisy speech features; and obtain single-frame noisy speech features of the short-time spectrum based on the frequency domain noisy speech features and the time domain noisy speech features.

[0043] According to one embodiment of the present disclosure, the training module is further used to: extract the initial noisy speech features of the short-time spectrum in the causal hole convolution layer of the feature extraction layer; and obtain any noisy speech features of the short-time spectrum based on the initial noisy speech features and the sigmoid activation function.

[0044] According to one embodiment of the present disclosure, the training module is further used to: perform a first feature extraction on the short-time spectrum input to the causal convolutional layer, extract the first frame feature of the speech frame corresponding to the short-time spectrum, and the noisy speech feature corresponding to the first frame feature; obtain the first noisy speech feature of the short-time spectrum based on the first frame feature and the noisy speech feature corresponding to the first frame feature; discard part of the first noisy speech feature, calculate the remaining features in the first noisy speech feature after discarding according to the ELU activation function, and use the calculated noisy speech feature as the initial noisy speech feature.

[0045] According to one embodiment of the present disclosure, the training module is further used to: perform a second feature extraction on the initial noisy speech feature, extract the second frame feature of the speech frame corresponding to the initial noisy speech feature, and the noisy speech feature corresponding to the second frame feature; obtain the second noisy speech feature of the short-time spectrum based on the second frame feature and the noisy speech feature corresponding to the second frame feature; perform a reference calculation on the second noisy speech feature according to the sigmoid activation function, and use the calculated noisy speech feature as the third noisy speech feature; normalize the second noisy speech feature and the third noisy speech feature, and use the normalized noisy speech feature as any noisy speech feature of the short-time spectrum output by the causal convolution layer.

[0046] According to one embodiment of the present disclosure, when the causal convolution layer outputs frequency domain noisy speech features, the causal void convolution layer is a frequency domain causal void convolution layer; when the causal convolution layer outputs frequency domain noisy speech features, the causal void convolution layer is a time domain causal void convolution layer.

[0047] According to one embodiment of the present disclosure, the training module is further used to: obtain single-frame speech data in single-frame multi-channel reverberation and noisy speech data, and obtain the scale-invariant signal-to-noise ratio loss and short-time objective intelligibility loss between the single-frame speech data and the single-frame denoised speech data; perform weighted summation of the scale-invariant signal-to-noise ratio loss and the short-time objective intelligibility loss to obtain the training loss function of the speech denoising model.

[0048] According to an embodiment of the present disclosure, the training module is further configured to cache the single-frame noisy speech feature as a historical single-frame noisy speech feature in a cache unit, waiting for the next call.

[0049] According to one embodiment of the present disclosure, the apparatus further includes: a distillation module configured to perform distillation processing on the first speech noise reduction model to obtain a lightweight second speech noise reduction model.

[0050] According to one embodiment of the present disclosure, the distillation module is further used to: obtain a lightweight speech denoising model to be trained based on the first speech denoising model; obtain the distillation loss of the lightweight speech denoising model based on the first speech denoising model based on the multi-channel reverberation and noisy speech data included in the training sample; adjust the parameters of the lightweight speech denoising model according to the distillation loss until the training is completed, and obtain the trained second speech denoising model.

[0051] According to one embodiment of the present disclosure, the distillation module is further used to: obtain lightweight extraction conditions of the model structure of the first speech denoising model; and construct a model to be trained whose model structure meets the lightweight extraction conditions as a lightweight speech denoising model.

[0052] According to one embodiment of the present disclosure, the distillation module is further used to: obtain the feature extraction loss and noise reduction loss of the lightweight speech denoising model based on the first speech denoising model according to the multi-channel reverberation and noisy speech data included in the training sample; and perform weighted summation of the feature extraction loss and the noise reduction loss to obtain the distillation loss of the lightweight speech denoising model based on the first speech denoising model.

[0053] According to one embodiment of the present disclosure, the distillation module is further used to: obtain a first noisy speech feature of the multi-channel reverberant noisy speech data included in the training samples extracted by the first speech denoising model; obtain a second noisy speech feature of the multi-channel reverberant noisy speech data included in the training samples extracted by the lightweight speech denoising model; and obtain a feature extraction loss of the lightweight speech denoising model based on the first speech denoising model based on the first noisy speech feature and the second noisy speech feature.

[0054] According to one embodiment of the present disclosure, the distillation module is further configured to: input the multi-channel reverberation noisy speech data included in the training sample into the first speech noise reduction model to obtain first noise-reduced speech output by the first speech noise reduction model; input the multi-channel reverberation noisy speech data included in the training sample into the lightweight speech noise reduction model to obtain second noise-reduced speech output by the lightweight speech noise reduction model; and obtain a noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model according to the first noise-reduced speech and the second noise-reduced speech.

[0055] According to one embodiment of the present disclosure, the reverberation module is further configured to: obtain source speech data and source noise data in the source noisy speech data; perform reverberation processing on the source speech data to obtain multi-channel reverberation speech data of the source speech data; perform reverberation processing on the source noise data to obtain multi-channel reverberation noise data of the source noise data; mix the multi-channel reverberation speech data and the multi-channel reverberation noise data to obtain multi-channel reverberation noisy speech data, and generate training samples according to the multi-channel reverberation noisy speech data.

[0056] According to one embodiment of the present disclosure, the reverberation module is further configured to: obtain initial speech data and initial noise data; perform data cleaning and data enhancement on the initial speech data and the initial noise data respectively to obtain source speech data corresponding to the initial speech data and source noise data corresponding to the initial noise data; and splice the source speech data and the source noise data to generate the source noisy speech data.

[0057] According to one embodiment of the present disclosure, the reverberation module is further configured to: adjust a speech length of the multi-channel reverberation noise data to be the same as a speech length of the multi-channel reverberation speech data; and mix the multi-channel reverberation noise data and the multi-channel reverberation speech data with the same speech length to obtain mixed multi-channel reverberation noisy speech data.

[0058] According to one embodiment of the present disclosure, the device further comprises an evaluation module configured to: obtain evaluation reverberation noisy speech data for evaluation according to the multi-channel reverberation noisy speech data; and evaluate the first speech noise reduction model and / or the second speech noise reduction model according to the evaluation reverberation noisy speech data.

[0059] According to one embodiment of the present disclosure, the evaluation module is further configured to: obtain evaluation parameters of the first speech noise reduction model and / or the second speech noise reduction model, and evaluate the first speech noise reduction model and / or the second speech noise reduction model according to the evaluation parameters.

[0060] To achieve the above-mentioned purpose, the fourth aspect of the present disclosure proposes a speech noise reduction device, which includes: an acquisition module, which is used to obtain multi-channel reverberant and noisy speech to be denoised, and perform short-time Fourier transform on the multi-channel reverberant and noisy speech data to be denoised frame by frame to generate a short-time spectrum to be denoised of a single frame of multi-channel reverberant and noisy speech data to be denoised; a noise reduction module, which is used to perform noise reduction processing on the short-time spectrum to be denoised frame by frame based on a trained target speech noise reduction model to obtain the denoised target speech data of the multi-channel reverberant and noisy speech data to be denoised output by the target speech noise reduction model.

[0061] In a fifth aspect, the present disclosure proposes an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute instructions to implement the training method of the noise reduction model proposed in the first aspect and / or the speech noise reduction method proposed in the second aspect.

[0062] In the sixth aspect of the present disclosure, a computer-readable storage medium is proposed. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the noise reduction model proposed in the first aspect and / or the speech noise reduction method proposed in the second aspect.

[0063] The training method and device of the noise reduction model proposed in the present disclosure obtain source noisy speech data, perform reverberation processing on the source noisy speech data, obtain multi-channel reverberation noisy speech data as samples after reverberation, and obtain training samples for training the speech noise reduction model based on multiple samples including multi-channel reverberation noisy speech data. Furthermore, the speech noise reduction model to be trained is trained by the multi-channel reverberation noisy speech data until the training is completed to obtain a trained first speech noise reduction model. In the present disclosure, the speech noise reduction model is trained by the multi-channel reverberation noisy speech data, so that the human ear has a good hearing experience for the noise reduction speech output by the trained speech noise reduction model, improves the transmission accuracy of the information carried in the speech data, and reduces the influence of noise on the transmission of the information carried in the speech data. The speech noise reduction model obtained by training with the multi-channel reverberation noisy speech data can be applied to multiple scenarios, improves the robustness of the speech noise reduction model, strengthens the applicability and practicality of the speech noise reduction model, and optimizes the training method and training effect of the speech noise reduction model.

[0064] It should be understood that the contents described in the present disclosure are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The above and / or additional aspects and advantages of the present disclosure will become apparent and more readily appreciated from the following description of embodiments, taken in conjunction with the accompanying drawings, in which:

[0066] Figure 1 A flowchart of a method for training a noise reduction model according to an embodiment of the present disclosure;

[0067] Figure 2 A flowchart of a method for training a noise reduction model according to another embodiment of the present disclosure;

[0068] Figure 3 A flowchart of a method for training a noise reduction model according to another embodiment of the present disclosure;

[0069] Figure 4 A flowchart of a method for training a noise reduction model according to another embodiment of the present disclosure;

[0070] Figure 5 A flowchart of a method for training a noise reduction model according to another embodiment of the present disclosure;

[0071] Figure 6 A flowchart of a method for training a noise reduction model according to another embodiment of the present disclosure;

[0072] Figure 7 A flowchart of a method for training a noise reduction model according to another embodiment of the present disclosure;

[0073] Figure 8 A schematic diagram of model evaluation according to an embodiment of the present disclosure;

[0074] Figure 9 A schematic diagram of model evaluation according to another embodiment of the present disclosure;

[0075] Figure 10 A schematic diagram of model evaluation according to another embodiment of the present disclosure;

[0076] Figure 11 A schematic diagram of model evaluation according to another embodiment of the present disclosure;

[0077] Figure 12 A flowchart of a speech noise reduction method according to an embodiment of the present disclosure;

[0078] Figure 13 A structural schematic diagram of a noise reduction model training apparatus according to an embodiment of the present disclosure;

[0079] Figure 14 A structural schematic diagram of a speech noise reduction apparatus according to an embodiment of the present disclosure;

[0080] Figure 15 A block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0081] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0082] The following describes a noise reduction model training method, speech noise reduction method, device, and electronic device proposed in embodiments of the present disclosure with reference to the accompanying drawings.

[0083] Figure 1 FIG. 1 is a flow chart of a method for training a noise reduction model according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0084] S101 , obtaining source noisy speech data, and performing reverberation processing on the source noisy speech data to generate corresponding training samples, wherein the training samples include multiple samples, each sample including multi-channel reverberation noisy speech data.

[0085] In implementation, when voice is used as a carrier for information transmission, the noise mixed in the voice will have a certain degree of impact on the accurate transmission of information. Therefore, in order to improve the information transmission based on the voice carrier, there is a need for noise reduction for the voice mixed with noise.

[0086] Herein, speech data mixed with noise may be determined as noisy speech data.

[0087] In the embodiment of the present disclosure, noise reduction processing of noisy speech data can be achieved by constructing a corresponding speech noise reduction model, wherein the noise reduction effect of the speech noise reduction model can be optimized by training the speech noise reduction model.

[0088] Optionally, the noisy speech data required for training the speech noise reduction model can be obtained based on the actual usage scenario of the speech noise reduction model. Furthermore, the noisy speech data required for training the speech noise reduction model can be obtained through real noisy speech data in actual scenarios.

[0089] Herein, real noisy speech data in an actual scene may be determined as the source noisy speech data.

[0090] In some implementations, the required source noisy speech data can be obtained through open speech data resources and open noise data resources on the Internet, such as the open speech data resource THCHS-30, the open speech data resource Primeword, the open speech data resource MAGICDATA, the open speech data resource Aishell, and the open noise data resource Wham_noise, the open noise data resource MUSAN_noise, and the open noise data resource DNS_noise.

[0091] In other implementations, the required source noisy speech data can be obtained by sound collection in actual scenarios. For example, relevant recording operations can be performed in an anechoic room to obtain corresponding speech data and noise data, thereby obtaining the required source noisy speech data.

[0092] In implementation, after the sound is generated by the sound source, it can propagate through the set medium and reflect on the objects in its propagation path. Therefore, in the set scenario, the sound heard by the human ear is the reverberation sound corresponding to the source noise sound generated by the sound source and the sound of the source noise sound reflected by the surrounding objects.

[0093] Therefore, in order to optimize the training effect of the speech noise reduction model and optimize the human ear's hearing experience of the noise-reduced speech output by the speech noise reduction model, the source noisy speech data can be reverberated, and the speech noise reduction model can be trained using the noisy speech data obtained after reverberation processing.

[0094] The noisy speech data obtained after the source noisy speech data is subjected to reverberation processing may be determined as the multi-channel reverberated noisy speech data.

[0095] Since the source noisy sound will produce multiple reflections in actual scenarios, the multiple reflected noisy speech data after the source noisy speech data is reflected by the object can be obtained through a set method, and mixed with the source noisy speech data to obtain multi-channel reverberated noisy speech data after the source noisy speech data is reverberated.

[0096] Furthermore, the multi-channel reverberant noisy speech data is used as samples for training the speech denoising model to be trained, and training samples corresponding to the speech denoising model to be trained are generated based on the multiple samples.

[0097] It can be understood that the training sample is a sample set consisting of multiple multi-channel reverberant and noisy speech data.

[0098] S102: Training a speech noise reduction model to be trained based on the multi-channel reverberant and noisy speech data included in the training samples to obtain a trained first speech noise reduction model.

[0099] In the embodiments of the present disclosure, the multi-channel reverberation noisy speech data included in the training sample can be input into the speech noise reduction model to be trained for model training until the training is completed.

[0100] Optionally, the plurality of multi-channel reverberation noisy speech data included in the training sample can be input into the speech noise reduction model to be trained in batches, and the speech noise reduction model can be trained according to the multi-channel reverberation noisy speech data input into the model in each batch.

[0101] The speech noise reduction model to be trained can be a time recurrent convolutional neural network model (TRCN), or can be a neural network model of other structures, which is not limited here.

[0102] Further, the multi-channel reverberation noisy speech data is stored in corresponding multi-channel reverberation noise-free speech data, so that the training label of the multi-channel reverberation noisy speech data can be generated according to the multi-channel reverberation noise-free speech data, and then the multi-channel reverberation noisy speech data is converted into the training sample of the speech noise reduction model, and is input into the speech noise reduction model to be trained for noise reduction model training.

[0103] Optionally, the corresponding training end condition can be set by the effect of model training. The model training result obtained after inputting the multi-channel reverberation noisy speech data as the training sample into the speech noise reduction model to be trained is compared with the set training end condition. When the training result output by a certain training round meets the set training end condition, it can be judged that the model after the training round can meet the needs of actual application, and then the training of the speech noise reduction model is completed.

[0104] Optionally, the corresponding training end condition can be set by the number of training rounds. The number of training rounds of the speech noise reduction model is monitored and recorded by the set method, and the number of training rounds recorded by the monitoring is compared with the set training end condition. When the number of training rounds corresponding to a certain round meets the set training end condition, it can be judged that the model after the training round can meet the needs of actual application, and then the training of the speech noise reduction model is completed.

[0105] Further, after the model training is completed, the trained speech noise reduction model is determined as the first speech noise reduction model.

[0106] The training method of the noise reduction model proposed in the present disclosure obtains source noisy speech data, performs reverberation processing on the source noisy speech data, obtains multi-channel reverberation noisy speech data as a sample after reverberation, and obtains training samples for training the speech noise reduction model based on multiple samples including the multi-channel reverberation noisy speech data. Furthermore, the speech noise reduction model to be trained is trained by the multi-channel reverberation noisy speech data until the training is completed to obtain a trained first speech noise reduction model. In the present disclosure, the speech noise reduction model is trained by the multi-channel reverberation noisy speech data, so that the human ear has a good hearing experience for the noise-reduced speech output by the trained speech noise reduction model, improves the propagation accuracy of the information carried in the speech data, and reduces the influence of noise on the propagation of the information carried in the speech data. The speech noise reduction model obtained by training with the multi-channel reverberation noisy speech data can be applied to multiple scenarios, improves the robustness of the speech noise reduction model, strengthens the applicability and practicality of the speech noise reduction model, and optimizes the training method and training effect of the speech noise reduction model.

[0107] In the above embodiment, the training of the speech noise reduction model can be combined with Figure 2 Further understanding, Figure 2 FIG. 1 is a flow chart of a method for training a noise reduction model according to another embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0108] S201 , performing short-time Fourier transform on multi-channel reverberant noisy speech data included in training samples frame by frame to generate a short-time spectrum of a single frame of multi-channel reverberant speech data.

[0109] In order to enable the trained first speech noise reduction model to achieve real-time noise reduction of the noisy speech data input therein, the multi-channel reverberant noisy speech data included in the training samples of the speech noise reduction model can be cut, and the speech noise reduction model can be trained with the cut multi-channel reverberant noisy speech data, so that the trained first speech noise reduction model can obtain the training effect of real-time noise reduction during the training process.

[0110] Optionally, the multi-channel reverberant and noisy speech data may be segmented based on the time dimension, wherein the multi-channel reverberant and noisy speech data may be segmented by frame, and each frame of the multi-channel reverberant and noisy speech data obtained after segmentation is determined as a single frame of multi-channel reverberant speech data.

[0111] Furthermore, in order to make the model have a better training effect, the single-frame multi-channel reverberation and noisy speech data can be converted into the corresponding short-time spectrum based on a set method.

[0112] Optionally, a short-time Fourier transform may be performed on the single-frame multi-channel reverberant and noisy speech data, and a short-time spectrum of the single-frame multi-channel reverberant and noisy speech data may be obtained based on the result of the short-time Fourier transform.

[0113] like Figure 3 As shown, the multi-channel reverberation and noisy speech data can be segmented by frames through the preprocessing module, and the single-frame multi-channel reverberation and noisy speech data obtained after segmentation can be short-time Fourier transformed to map the single-frame multi-channel reverberation and noisy speech data to the corresponding short-time spectrum domain, thereby generating the short-time spectrum corresponding to the segmented single-frame multi-channel reverberation speech data.

[0114] S202: Train the speech noise reduction model to be trained according to the short-time spectrum to obtain a trained first speech noise reduction model.

[0115] In the embodiment of the present disclosure, the short-time spectrum output by the preprocessing module can be input into the input end of the speech denoising model to be trained, and the speech denoising model to be trained can be trained using the short-time spectrum corresponding to the single-frame multi-channel reverberation-noised speech data.

[0116] Optionally, time-frequency masking may be performed on the short-time spectrum to obtain a spectrum mask of the short-time spectrum.

[0117] In implementation, the single-frame multi-channel reverberation and noisy speech data can be analyzed according to the masking effect of the sound, and the single-frame multi-channel reverberation speech data and the single-frame multi-channel reverberation noise data included in the single-frame multi-channel reverberation and noisy speech data can be used as masking sounds, and the single-frame source speech data included in the single-frame multi-channel reverberation and noisy speech data can be used as masked sounds.

[0118] Furthermore, based on the set parameters of the masker sound and the masked sound, time-frequency masking is performed on the short-time spectrum corresponding to the single-frame multi-channel reverberant noisy speech data in the time domain and the frequency domain, thereby obtaining a masked spectrum mask.

[0119] In the embodiment of the present disclosure, time-frequency masking of the short-time spectrum can be implemented based on the speech noise reduction model, and then the spectrum mask after time-frequency masking of the short-time spectrum can be obtained from the intermediate layer output of the speech noise reduction model.

[0120] In order to achieve time-frequency masking of the short-time spectrum based on the speech noise reduction model, the speech noise reduction model can be used to extract features of the short-time spectrum input therein, obtain the masking features and masked features in the features included in the short-time spectrum, and then obtain the spectrum mask corresponding to the short-time spectrum from the intermediate layer output of the speech noise reduction model.

[0121] Among them, the feature extraction layer of the speech noise reduction model can be used to extract the single-frame noisy speech features of the short-time spectrum.

[0122] Furthermore, the speech denoising model has a feature extraction layer, which can perform feature extraction on the short-time spectrum to obtain the noisy speech features in the single-frame multi-channel reverberation noisy speech data carried in the short-time spectrum, and determine it as the single-frame noisy speech features of the short-time spectrum.

[0123] In order to achieve time-frequency masking of the short-time spectrum, the noisy speech features in the time domain and frequency domain dimensions of the single-frame multi-channel reverberation noisy speech data can be extracted, and the single-frame noisy speech features extracted by the feature extraction layer of the short-time spectrum can be obtained based on the noisy speech features in the time domain and frequency domain dimensions.

[0124] It should be noted that the feature extraction layer has a set cache unit, which can cache the single-frame noisy speech features extracted by the feature extraction layer as historical single-frame noisy speech features in the cache unit to wait for the next call.

[0125] Among them, the cache unit can be set at a set position in the feature extraction layer. When the feature extraction layer extracts a single-frame noisy speech feature corresponding to the short-time spectrum input therein, it can be cached in the cache unit at the set position and saved as a historical single-frame noisy speech feature.

[0126] Optionally, there is a set calling condition for the historical single-frame noisy speech feature. When the calling condition is met, the cache unit can be accessed and the corresponding historical single-frame noisy speech feature can be called.

[0127] Among them, by accessing and calling the historical single-frame noisy speech features in the cache unit, the single-frame noisy speech features extracted by the feature extraction layer can maintain the time sequence information.

[0128] Furthermore, the noisy speech features of the short-time spectrum in the frequency domain dimension can be determined as the frequency domain noisy speech, and the noisy speech features of the short-time spectrum in the time domain dimension can be determined as the time domain noisy speech features.

[0129] Furthermore, a feature extraction layer of a speech denoising model can be constructed based on causal dilated convolution, and frequency domain features of the short-time spectrum can be extracted based on the frequency domain causal dilated convolution layer in the feature extraction layer to obtain frequency domain noisy speech features.

[0130] like Figure 3 As shown in Figure 1, the short-time spectrum can be input into the frequency-domain causal dilated convolution layer, and the frequency-domain causal dilated convolution layer can be used to perform dilated convolution on the short-time spectrum in the frequency domain.

[0131] In implementation, the noise of the single-frame multi-channel reverberation noisy speech data can be distributed in the entire frequency dimension or concentrated in a narrow band, and therefore, the receptive field can be controlled through the hole convolution in the frequency dimension to achieve the cross-correlation between the respective fundamental frequencies, thereby extracting the frequency domain noisy speech features of the short-time spectrum in the frequency dimension, wherein the frequency domain noisy speech features can include the fundamental frequency features of the single-frame multi-channel reverberation noisy speech data in the frequency dimension.

[0132] It should be noted that the hole convolution step in the frequency domain causal hole convolution layer can be set as 2l, l is a positive integer, and at least one frequency domain causal hole convolution layer can be set in the speech noise reduction model as a feature extraction layer for extracting the frequency domain noisy speech features corresponding to the short-time spectrum.

[0133] Correspondingly, the time domain causal hole convolution layer for extracting the time domain features in the speech noise reduction model can be constructed based on the causal hole convolution, wherein the short-time spectrum is subjected to time domain feature extraction by the time domain causal hole convolution layer in the feature extraction layer to obtain the time domain noisy speech features.

[0134] As shown in Figure 3 , the short-time spectrum can be input into the time domain causal hole convolution layer, and the short-time spectrum is subjected to hole convolution in the time domain dimension by the time domain causal hole convolution layer, thereby obtaining the time domain noisy speech features of the short-time spectrum in the time domain dimension.

[0135] It should be noted that, in the single-frame multi-channel noisy speech data, the duration of the single-frame multi-channel noisy data is relatively shorter than that of the single-frame multi-channel reverberation speech data without noise, and therefore, the step length of the time domain causal hole convolution layer is smaller than that of the frequency domain causal hole convolution layer.

[0136] In order to better understand the feature extraction process of the frequency domain noisy speech features and the time domain noisy speech features, the process of obtaining any noisy speech feature therebetween can be combined with the following example:

[0137] Optionally, the initial noisy speech features of the short-time spectrum can be extracted in the causal hole convolution layer of the feature extraction layer.

[0138] In the embodiments of the present disclosure, the causal hole convolution structure corresponding to the feature extraction layer of the speech noise reduction model can be as shown in Figure 4 , and the short-time spectrum can be subjected to feature extraction by the causal hole convolution layer shown in Figure 4 , thereby obtaining the noisy speech features in the corresponding dimension.

[0139] It should be noted that when the causal convolution layer outputs the frequency domain noisy speech features, the causal hole convolution layer is a frequency domain causal hole convolution layer. When the causal convolution layer outputs the time domain noisy speech features, the causal hole convolution layer is a time domain causal hole convolution layer.

[0140] Furthermore, a first feature extraction is performed on the short-time spectrum input to the causal convolutional layer to extract the first frame feature of the speech frame corresponding to the short-time spectrum and the noisy speech feature corresponding to the first frame feature.

[0141] like Figure 4 As shown, the convolution unit 1 can perform the first feature extraction inside the causal convolution layer on the short-time spectrum of the input time domain causal convolution layer or the frequency domain causal convolution layer to obtain the features of the speech frame corresponding to the short-time spectrum and the noisy speech features corresponding to the speech frame features.

[0142] The frame feature of the speech frame obtained by the first feature extraction may be determined as the first frame feature, and the noisy speech feature corresponding to the speech frame feature is the noisy speech feature corresponding to the first frame feature.

[0143] Furthermore, a first noisy speech feature of the short-time spectrum is obtained according to the first frame feature and the noisy speech feature corresponding to the first frame feature.

[0144] like Figure 4 As shown, the first frame feature and the large noisy speech feature corresponding to the first frame feature can be fused under the set method in the convolution unit 1, so as to obtain a speech feature that carries both the first frame feature and the noisy speech feature corresponding to the first frame feature, and determine it as the first noisy speech feature.

[0145] In order to optimize the training effect of the speech noise reduction model, you can also Figure 4 The first noisy speech feature output by the convolution unit 1 shown is partially discarded by a random dropout unit, the remaining features in the discarded first noisy speech feature are calculated according to the ELU activation function, and the calculated noisy speech feature is used as the initial noisy speech feature.

[0146] Furthermore, the initial noisy speech features are input into Figure 4 The convolution unit 2 and the convolution unit 3 shown in the figure obtain any noisy speech feature of the short-time spectrum according to the initial noisy speech feature and the sigmoid activation function.

[0147] The initial noisy speech feature can be subjected to a second feature extraction process using convolution units 2 and 3 to extract a second frame feature of the speech frame corresponding to the initial noisy speech feature and a noisy speech feature corresponding to the second frame feature. A second noisy speech feature of the short-time spectrum is obtained based on the second frame feature and the noisy speech feature corresponding to the second frame feature.

[0148] Optionally, the initial noisy speech feature can be downsampled by convolution unit 2 and convolution unit 3 respectively to obtain a second frame feature of the speech frame corresponding to the initial noisy speech feature and a noisy speech feature corresponding to the second frame feature.

[0149] Furthermore, in convolution unit 2 and convolution unit 3, feature fusion is performed on the second frame feature and the noisy speech feature corresponding to the second frame feature based on the set method, thereby obtaining the noisy speech feature obtained by downsampling the initial noisy speech, and identifying it as the second noisy speech feature.

[0150] Accordingly, if Figure 4 As shown, the second noisy speech feature output by the convolution unit 3 can also be referenced and calculated according to the sigmoid activation function, and the calculated noisy speech feature can be used as the third noisy speech feature.

[0151] like Figure 4 As shown, the second noisy speech feature and the third noisy speech feature can be normalized by the normalization layer, and the noisy speech feature obtained by normalization is used as the output result of the normalization layer, and is used as any one of the noisy speech features in the time domain noisy speech feature and the frequency domain noisy speech feature of the short time spectrum output by the causal convolution layer.

[0152] Furthermore, single-frame noisy speech features of the short-time spectrum are obtained based on the frequency-domain noisy speech features and the time-domain noisy speech features.

[0153] Optionally, based on the set feature fusion layer of the speech noise reduction model, the time domain noisy speech features and the frequency domain noisy speech features can be fused to obtain noisy speech features that carry both time domain dimension features and frequency domain dimension features, and used as the single-frame noisy speech features of the short-time spectrum corresponding to the single-frame multi-channel reverberation noisy speech data output after feature extraction by all feature extraction layers of the speech noise reduction model.

[0154] Furthermore, by performing time-frequency masking on the single-frame noisy speech features, video masking of the short-time spectrum is achieved, and then a spectrum mask corresponding to the short-time spectrum is obtained.

[0155] Among them, the masking parameters of the single-frame noisy speech feature can be obtained, and the single-frame noisy speech feature is subjected to time-frequency masking according to the masking parameters to obtain a spectrum mask. Figure 3 As shown, the single-frame noisy speech feature can be input Figure 3 The gated linear unit in Figure 3 The gated linear unit in performs time-frequency masking on the noisy speech features of a single frame, thereby obtaining a spectrum mask of the short-time spectrum, which is used as the output result of the gated linear unit.

[0156] Optionally, the masking parameter of the single-frame noisy speech feature may be determined based on the distribution of the noise-free source speech, the distribution of the noise-free reverberant speech, and the corresponding noise distribution in the single-frame noisy speech feature.

[0157] According to the obtained masking parameters, time-frequency masking is performed on the features corresponding to the noiseless reverberation speech and the noise in the single-frame noisy speech features to obtain the speech features corresponding to the noiseless source speech, and then the spectrum mask corresponding to the short-time spectrum is obtained.

[0158] It should be noted that the speech noise reduction model may include a set number of unidirectional gated linear units, such as 2, or may include other numbers of other types of gated linear units, which is not limited here.

[0159] Furthermore, according to the spectrum mask and the short-time spectrum, a denoised short-time spectrum is obtained after the short-time spectrum is denoised.

[0160] In the implementation, in order to avoid the influence of the feature loss of single-frame noisy speech features in single-frame reverberant noisy speech data on the model training effect, the short-time spectrum and its corresponding spectrum mask can be calculated based on the set algorithm, and the speech spectrum after denoising of the single-frame multi-channel reverberant noisy speech data corresponding to the short-time spectrum can be obtained from the calculation result, and it is determined as the denoised short-time spectrum after the short-time spectrum denoising.

[0161] Optionally, a residual connection exists in the sound noise reduction model, and the short-time spectrum can be transmitted to a set position through the set residual connection, and the short-time spectrum and spectrum mask are calculated at the set position, thereby obtaining a denoised short-time spectrum after the short-time spectrum is denoised.

[0162] like Figure 3 As shown, the short-time spectrum and the spectrum mask can be processed by the time-domain causal hole deconvolution layer according to the set algorithm to obtain the denoised short-time spectrum after the short-time spectrum is denoised, and the denoised short-time spectrum is used as the output result of the time-domain causal hole deconvolution layer.

[0163] Therefore, based on Figure 3 The residual connection set as shown transmits the short-time spectrum of the input time-domain causal hole convolution layer to the time-domain causal hole deconvolution layer, thereby realizing the acquisition of the denoised short-time spectrum in the time-domain causal hole deconvolution layer.

[0164] Furthermore, the denoised short-time spectrum may be subjected to a short-time inverse Fourier transform to generate speech data of the denoised short-time spectrum, which serves as single-frame denoised speech data outputted by the training of the speech denoising model.

[0165] Since the short-time spectrum is obtained based on the short-time Fourier transform of the single-frame multi-channel reverberant and noisy speech data, the denoised short-time spectrum obtained after the short-time spectrum is denoised can be subjected to an inverse short-time Fourier transform to obtain the denoised speech data of the single-frame multi-channel reverberant and noisy speech data corresponding to the short-time spectrum.

[0166] The speech data after denoising the single-frame multi-channel reverberation-noised speech data can be determined as the single-frame denoised speech data, and used as the training output result of the speech denoising model.

[0167] Optionally, you can pass Figure 3 The post-processing module shown performs short-time inverse Fourier transform on the denoised short-time spectrum, thereby obtaining single-frame denoised speech data output by the training of the speech denoising model.

[0168] For example, set the short-time spectrum of the input speech denoising model M1 to be trained as xf. After inputting xf into M1, the corresponding spectrum mask m is obtained. The denoised short-time spectrum sf is obtained according to xf and m. The sf is subjected to an inverse short-time Fourier transform to obtain the single-frame denoised speech data s after denoising the single-frame multi-channel reverberant noisy speech data corresponding to xf.

[0169] Furthermore, a training loss function of the speech noise reduction model is obtained based on the single-frame denoised speech data, and parameters of the speech noise reduction model are adjusted according to the training loss function until the training is completed, thereby obtaining a trained first speech noise reduction model.

[0170] In the embodiment of the present disclosure, the single-frame multi-channel reverberated noisy speech input to the speech denoising model for training includes single-frame speech data without noise and reverberation. Therefore, the optimized training of the speech denoising model can be achieved based on the loss between the single-frame denoised speech data output by the training and the single-frame speech data input to the model for training.

[0171] Among them, the loss between the single-frame denoised speech data output by training and the single-frame speech data input to the model for training can be obtained by calculating the loss function between the single-frame denoised speech data and the single-frame speech data.

[0172] Optionally, single-frame speech data in the single-frame multi-channel reverberation noisy speech data can be obtained, and a scale-invariant signal-to-noise ratio (Si-SNR) loss and a short-time objective intelligibility (STOI) loss between the single-frame speech data and the single-frame denoised speech data can be obtained.

[0173] The loss between single-frame denoised speech data and single-frame speech data is measured by obtaining the scale-invariant signal-to-noise ratio loss between single-frame speech data and single-frame denoised speech data and the short-term objective intelligibility loss between single-frame speech data and single-frame denoised speech data.

[0174] Further, the scale-invariant signal-to-noise ratio loss and the short-time objective intelligibility loss can be weighted and summed to obtain a training loss function of the speech denoising model.

[0175] The scale-invariant signal-to-noise ratio loss and the short-time objective intelligibility loss are weighted and summed by a set weight ratio.

[0176] For example, the weight ratio of the scale-invariant signal-to-noise ratio loss is set to 0.3, and the weight ratio of the short-time objective intelligibility loss is set to 0.7. The obtained scale-invariant signal-to-noise ratio loss and short-time objective intelligibility loss are weighted and summed based on the set weight ratio, and then a loss function between the single-frame denoised speech data and the single-frame speech data is obtained. The loss function is used as the training loss function of the speech denoising model.

[0177] Further, the model parameters of the speech denoising model are adjusted according to the training loss function obtained in each training round. Optionally, the model parameters can be adjusted by gradient descent method. For example, the learning rate of the gradient descent method is set to 0.003 to adjust the model parameters of the speech denoising model.

[0178] It should be noted that in the scenario where the learning rate does not decrease in two consecutive model training rounds, the set learning rate can be adjusted to decrease.

[0179] Further, when the speech denoising model meets the set training end condition, the model training is ended, and the model obtained after the last training round is ended is used as the trained first speech denoising model.

[0180] Optionally, whether the speech denoising model trained in the current round meets the set training end condition can be determined based on the output result of the training round. For example, the condition that the learning rate does not decrease in N consecutive training rounds is set as the end condition of the model training. The learning rate corresponding to each round of model training can be compared. If the learning rate corresponding to N consecutive rounds of model training does not decrease after the model training in the current round is ended, it is determined that the speech denoising model trained in the current round meets the training end condition. The speech denoising model after the current round of training can be used as the trained first speech denoising model.

[0181] Optionally, it can be judged whether the voice noise reduction model after the current round of training meets the set training end condition based on the round of training. For example, if 60 rounds of training are set as the training end condition of the model, the rounds of model training can be monitored and recorded. If the recorded training rounds of the model after the current round of model training meet 60 rounds, it can be determined that the model after the current round of training meets the training end condition. Therefore, the voice noise reduction model after the current round of training can be used as the trained first voice noise reduction model.

[0182] It should be noted that the last N rounds of training weight files in the voice noise reduction model training process can be obtained and integrated to obtain the weight file corresponding to the trained first voice noise reduction model. In the scenario where the first voice noise reduction model needs to be called, the model structure of the first voice noise reduction model can be built and the corresponding weight file can be loaded to realize the calling of the first voice noise reduction model.

[0183] The weight file can be kept in a set format, such as.pth format.

[0184] The training method of the noise reduction model proposed in the present disclosure is to divide the multi-channel reverberation and noisy speech data into single-frame multi-channel reverberation and noisy speech data by frame, and perform short-time Fourier transform on the data to obtain the short-time spectrum corresponding to the single-frame multi-channel reverberation and noisy speech data. The speech noise reduction model is trained by the short-time spectrum until the training is completed, wherein the time domain noisy speech features and frequency domain noisy speech features of the short-time spectrum are extracted by the feature extraction layer of the speech noise reduction model, and then the single-frame noisy speech features corresponding to the short-time spectrum are obtained. Furthermore, the single-frame noisy speech features are subjected to time-frequency masking, thereby achieving time-frequency masking of the short-time spectrum, and then the corresponding spectrum mask is obtained. According to the short-time spectrum and its corresponding spectrum mask, the denoised short-time spectrum after the short-time spectrum is denoised is obtained, and the short-time inverse Fourier transform is performed on the short-time spectrum to obtain the single-frame denoised speech data corresponding to the denoised short-time spectrum, and the denoised short-time spectrum is used as the training output result of the speech noise reduction model. Furthermore, a training loss function of the speech noise reduction model is obtained, and the model parameters of the speech noise reduction model are adjusted according to the training loss function until the training is completed, thereby obtaining a trained first speech noise reduction model. In the present disclosure, the speech noise reduction model is trained by multi-channel reverberation and noisy speech data, so that the human ear has a good hearing experience for the noise-reduced speech output by the trained speech noise reduction model, the propagation accuracy of the information carried in the speech data is improved, and the influence of noise on the propagation of the information carried in the speech data is reduced. The multi-channel reverberation and noisy speech data is framed, and the speech noise reduction model is trained by the short-time spectrum corresponding to the single-frame multi-channel reverberation and noisy speech data, and the training method of the model is optimized, so that the trained speech noise reduction model realizes real-time noise reduction of noisy speech, and the applicability and practicality of the speech noise reduction model are optimized.

[0185] In order to realize the multi-scenario application of the first speech noise reduction model, the first speech noise reduction model can be distilled to obtain a lightweight second speech noise reduction model, wherein the lightweight speech noise reduction model to be trained can be constructed based on the first speech noise reduction model, and trained by the first speech noise reduction model until the training is completed to obtain the trained lightweight second speech noise reduction model.

[0186] Among them, the acquisition of the lightweight second speech noise reduction model corresponding to the first speech noise reduction model can be combined with Figure 5 Further understanding, Figure 5 FIG. 1 is a flow chart of a method for training a noise reduction model according to another embodiment of the present disclosure. Figure 5 As shown, the method includes:

[0187] S501: Obtain a lightweight speech noise reduction model to be trained according to the first speech noise reduction model.

[0188] During implementation, there are set requirements for the hardware device that carries the first speech noise reduction model. Therefore, in order to realize the multi-scenario application of the first speech noise reduction model, especially the application on micro or small hardware devices, a lightweight speech noise reduction model corresponding to the first speech noise reduction model can be obtained and identified as a lightweight speech noise reduction model.

[0189] Among them, the first speech noise reduction model has a set lightweight extraction condition, and the lightweight extraction condition of the model structure of the first speech noise reduction model can be obtained, and according to the lightweight extraction condition, a model to be trained whose model structure meets the lightweight extraction condition is constructed as a lightweight speech noise reduction model.

[0190] Among them, a corresponding lightweight speech noise reduction model can be constructed according to the lightweight extraction conditions.

[0191] For example, the lightweight extraction conditions are set as follows: based on the model structure of the first speech denoising model, the number of causal hole convolution layers is one quarter of the number of causal hole convolution layers in the first speech denoising model, and the number of causal hole deconvolution layers is one quarter of the number of causal hole deconvolution layers in the first speech denoising model.

[0192] The number of time-domain causal hole convolution layers can be reduced to one-quarter, the number of frequency-domain causal hole convolution layers can be reduced to one-quarter, and the number of time-domain causal hole deconvolution layers can be reduced to one-quarter. Based on the model structure of the first speech noise reduction model, a corresponding lightweight noise reduction model to be trained is constructed and determined as the lightweight speech noise reduction model to be trained corresponding to the first speech noise reduction model.

[0193] S502 : Obtaining a distillation loss of a lightweight speech denoising model based on a first speech denoising model according to the multi-channel reverberant noisy speech data included in the training sample.

[0194] In an embodiment of the present disclosure, multi-channel reverberant noisy speech data included in the same training sample can be input into a trained first speech denoising model and a lightweight speech denoising model to be trained. The distillation loss of the lightweight speech denoising model based on the first speech denoising model is obtained through the output results of the first speech denoising model and the output results of the lightweight speech denoising model to be trained.

[0195] Optionally, based on the feature extraction dimension and the speech noise reduction dimension, the feature extraction loss and noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model can be obtained according to the multi-channel reverberation and noisy speech data included in the training sample.

[0196] The acquisition of the feature extraction loss of the lightweight speech noise reduction model based on the first speech noise reduction model can be combined with the following examples:

[0197] A first noisy speech feature of the multi-channel reverberant noisy speech data included in the training sample extracted by the first speech noise reduction model is obtained.

[0198] The multi-channel reverberant and noisy speech data included in the training sample can be input into the first speech denoising model, and the feature extraction layer of the first speech denoising model can be used to extract noisy speech features from the multi-channel reverberant and noisy speech data input therein.

[0199] The noisy speech features extracted by the feature extraction layer of the first speech noise reduction model may be identified as first noisy speech features.

[0200] Accordingly, the second noisy speech feature of the multi-channel reverberant noisy speech data included in the training sample extracted by the lightweight speech denoising model is obtained.

[0201] Furthermore, the multi-channel reverberant noisy speech data included in the training samples input into the first speech denoising model can be input into the lightweight speech denoising model to be trained, and the noisy speech features therein can be extracted through the feature extraction layer of the lightweight speech denoising model.

[0202] The noisy speech feature extracted by the feature extraction layer of the lightweight speech denoising model may be identified as the second noisy speech feature.

[0203] Furthermore, according to the first noisy speech feature and the second noisy speech feature, a feature extraction loss of the lightweight speech denoising model based on the first speech denoising model is obtained.

[0204] Among them, the first noisy speech feature is the feature extracted by the trained first speech noise reduction model, and the second noisy speech feature is the feature extracted by the lightweight speech noise reduction model to be trained. Therefore, by obtaining the loss between the first noisy speech feature and the second noisy speech feature, the loss of the lightweight speech noise reduction model based on the first speech noise reduction model in the feature extraction dimension can be obtained.

[0205] The loss of the lightweight speech noise reduction model based on the first speech noise reduction model in the feature extraction dimension can be determined as the feature extraction loss of the lightweight speech noise reduction model based on the first speech noise reduction model.

[0206] Optionally, a feature distance between the first noisy speech feature and the second noisy speech feature can be obtained, and the loss between the first noisy speech feature and the second noisy speech feature can be measured by the feature distance between the two, thereby obtaining the feature extraction loss of the lightweight speech denoising model based on the first speech denoising model.

[0207] For example, setting the first noisy speech feature as Ft and the second noisy speech feature as Fs, the feature extraction loss of the lightweight speech denoising model based on the first speech denoising model can be obtained by obtaining the loss between Ft and Fs.

[0208] Accordingly, the acquisition of the noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model can be combined with the following examples:

[0209] The multi-channel reverberant noisy speech data included in the training sample is input into the first speech noise reduction model to obtain the first noise-reduced speech output by the first speech noise reduction model.

[0210] Optionally, the multi-channel reverberant and noisy speech data included in the training sample can be input into the first speech noise reduction model, and the reverberant speech data and the noise speech data in the multi-channel reverberant and noisy speech data can be masked by the first speech noise reduction model to obtain the speech data after the multi-channel reverberant and noisy speech data is denoised.

[0211] The speech data after noise reduction of the multi-channel reverberant and noisy speech data may be identified as the first noise-reduced speech.

[0212] Accordingly, the multi-channel reverberant noisy speech data included in the training sample is input into the lightweight speech noise reduction model to obtain the second noise-reduced speech output by the lightweight speech noise reduction model.

[0213] Among them, the multi-channel reverberant and noisy speech data included in the training samples input into the first speech noise reduction model can be input into the lightweight speech noise reduction model to be trained, and the speech data after the lightweight speech noise reduction model performs noise reduction on the multi-channel reverberant and noisy speech data can be obtained from the output result of the lightweight speech noise reduction model, and it is identified as the second noise-reduced speech output by the lightweight speech noise reduction model.

[0214] Furthermore, according to the first noise reduction speech and the second noise reduction speech, a noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model is obtained.

[0215] Optionally, the first denoised speech is the speech data obtained after the first speech denoising model performs denoising on the multi-channel reverberant and noisy speech data, and the second denoised speech is the speech data obtained after the lightweight speech denoising model performs denoising on the multi-channel reverberant and noisy speech data. Therefore, by obtaining the loss between the first denoised speech and the second denoised speech, the loss of the lightweight speech denoising model and the first speech denoising model in the noise reduction dimension can be obtained.

[0216] The loss of the lightweight speech noise reduction model and the first speech noise reduction model in the noise reduction dimension is determined as the noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model.

[0217] Furthermore, a weighted sum is performed on the feature extraction loss and the denoising loss to obtain the distillation loss of the lightweight speech denoising model based on the first speech denoising model.

[0218] In order to optimize the training effect of the lightweight speech denoising model, the feature extraction loss and noise reduction loss of the obtained lightweight speech denoising model based on the first speech denoising model can be weightedly summed, and according to the result of the weighted summation, the distillation loss of the lightweight speech denoising model based on the first speech denoising model can be obtained.

[0219] S503: Adjust parameters of the lightweight speech denoising model according to the distillation loss until the training is completed, and obtain a trained second speech denoising model.

[0220] In the embodiment of the present disclosure, the model parameters of the lightweight speech denoising model can be adjusted based on the distillation loss of the first speech denoising model obtained until the training is completed.

[0221] Optionally, the model parameters of the lightweight speech denoising model may be adjusted and optimized based on a gradient descent method. For example, the model parameters of the lightweight speech denoising model may be adjusted and optimized based on a learning rate of 0.001 using the gradient descent method.

[0222] Furthermore, the training end conditions of the lightweight speech noise reduction model can be obtained, and the training process of the lightweight speech noise reduction model can be monitored. When it is monitored that the lightweight speech noise reduction model meets the training end conditions, the training is ended and the lightweight speech noise reduction model obtained at the end of the last round of training is determined as the lightweight second speech noise reduction model of the trained first speech noise reduction model.

[0223] It should be noted that, during the training process of the lightweight speech denoising model, the training weight files of the last N rounds may be integrated to obtain the weight file corresponding to the second speech denoising model.

[0224] Optionally, after the lightweight speech noise reduction model is trained to obtain the second speech noise reduction model, the second speech noise reduction model may be quantized using the setting method, thereby further reducing the storage space required by the second speech noise reduction model.

[0225] Optionally, the weight file of the second speech noise reduction model can also be quantized using a set quantization method. For example, the weight file of the second speech noise reduction model can be quantized using a dynamic asymmetric quantization method, thereby effectively reducing the storage space required by the weight file of the second speech noise reduction model.

[0226] Furthermore, in a scenario where the second speech noise reduction model needs to be called, the model structure of the second speech noise reduction model can be constructed and its corresponding weight file can be loaded, thereby realizing the calling of the second speech noise reduction model.

[0227] The training method of the noise reduction model proposed in the present disclosure constructs a lightweight lightweight speech noise reduction model to be trained based on the first speech noise reduction model, uses the first speech noise reduction model as a teacher model, and trains the lightweight speech noise reduction model through multi-channel reverberation and noisy speech data, so that the lightweight speech noise reduction model can learn the first speech noise reduction model, thereby obtaining a lightweight second speech noise reduction model corresponding to the first speech noise reduction model. In the present disclosure, a lightweight second speech noise reduction model is obtained by model distillation of the first speech noise reduction model, so that the speech noise reduction model can be deployed on small terminals and micro terminals, thereby improving the applicability and practicality of the speech noise reduction model.

[0228] In the above embodiment, the acquisition of multi-channel reverberation and noisy speech data required for model training can be combined with Figure 6 Further understanding, Figure 6 FIG. 1 is a flow chart of a method for training a noise reduction model according to another embodiment of the present disclosure. Figure 6 As shown, the method includes:

[0229] S601: Acquire source speech data and source noise data from source noisy speech data.

[0230] In the embodiment of the present disclosure, the sounds in the actual usage scenarios of the trained speech noise reduction model include multi-channel reverberant and noisy sounds. Therefore, when training the speech noise reduction model, it is necessary to simulate the actual usage scenarios and train the model through multi-channel reverberant and noisy speech data.

[0231] Optionally, noise-free and reverberation-free speech data may be obtained from the source noisy speech data as the source speech data, and correspondingly, noise-free and reverberation-free noise data may be obtained from the source noisy speech data as the source noise data.

[0232] The following examples can be used to obtain source noisy speech data:

[0233] Optionally, initial speech data and initial noise data may be acquired.

[0234] Initial speech data can be obtained by sampling from an open-source speech dataset on the Internet, or by performing speech simulation in a noise-free environment and collecting speech data to obtain the corresponding speech data.

[0235] Accordingly, initial noise data can be obtained by sampling from an open source noise dataset on the Internet, or by performing noise simulation in a set noise simulation environment and collecting noise data to obtain the corresponding initial noise data.

[0236] Furthermore, data cleaning and data enhancement may be performed on the initial speech data and the initial noise data, respectively, to obtain source speech data corresponding to the initial speech data and source noise data corresponding to the initial noise data.

[0237] In the embodiment of the present disclosure, the acquired initial voice data and initial noise data may contain unusable data such as silence data, voice data with too short a duration, and noise data with too short a duration. Therefore, the acquired initial voice data and initial noise data can be cleaned separately, and the silence data, voice data with a duration less than a set duration threshold, and noise data with a duration less than a set duration threshold are all deleted to obtain the initial voice data and initial noise data after data cleaning.

[0238] Furthermore, data enhancement may be performed on the initial speech data and the initial noise data after data cleaning, so as to enhance the speech features corresponding to the initial speech data and the initial noise data, respectively.

[0239] Optionally, the data enhancement method of the speech data may include frequency domain masking, time domain masking, speed transformation, and amplitude clipping.

[0240] Among them, in the scenario of using frequency domain mask for data enhancement, some frequency bands in the speech can be removed by notch filtering to achieve small-scale destruction of signal frequency bands, thereby achieving data enhancement of the filtered collected speech data.

[0241] When using a time domain mask for data augmentation, some sampling points in the time domain corresponding to the filtered collected speech data can be directly discarded, thereby achieving data augmentation for the filtered collected speech data. Some sampling points can be replaced with set null values ​​or with uniformly distributed noise that follows set noise parameters, without limitation here.

[0242] In the scenario where speed conversion is used for data enhancement, data enhancement of the filtered collected speech data can be achieved through resampling.

[0243] In a scenario where amplitude clipping is used for data enhancement, the speech intensity of the filtered collected speech data can be clipped to be within the absolute value range corresponding to the set threshold, thereby achieving data enhancement of the filtered collected speech data.

[0244] Correspondingly, the initial noise data after data cleaning can be data enhanced based on the above method of data enhancing the initial voice data after data cleaning, and the specific process can be referred to the above related content, which will not be repeated here.

[0245] Further, the source voice data and the source noise data are spliced to generate source noisy voice data.

[0246] In the embodiments of the present disclosure, the source voice data and the source noise data can be spliced, and the spliced data is determined as the source noisy voice data.

[0247] Optionally, the source voice data and the source noise data can be spliced and integrated based on the same frequency, and then the integrated source noisy voice data is obtained.

[0248] S602, reverberation processing is performed on the source voice data to obtain multi-channel reverberation voice data of the source voice data.

[0249] In the embodiments of the present disclosure, the sound of the sound reflected by the object in the actual scene can be simulated by setting the method, and then the reverberation processing of the source voice data is realized.

[0250] Optionally, the multi-channel reverberation voice data corresponding to the source voice data can be constructed based on the mirror method.

[0251] Among them, according to the object attribute information in the actual use scene, the reflection of the source voice on the object is simulated to obtain the voice data reflected by the source voice data.

[0252] For example, the room size, the reflection coefficient of the wall, the room T60, the position parameters of the sound collecting device and the position parameters of the sound source and other related object attribute information of the actual use scene can be combined to randomly sample from the room parameter distribution, and the corresponding sound decay time is estimated by using the room T60.

[0253] Among them, the order of the mirror method is estimated according to the size of the room, and the source voice data impulse response is simulated and generated, and then the voice data reflected by the source voice data on the object is obtained according to the source voice data impulse response.

[0254] Further, the voice data reflected by the source voice data and the source voice data can be mixed, and the mixed voice data is obtained as the multi-channel reverberation voice data after the source voice data is reverberation processed.

[0255] S603, reverberation processing is performed on the source noise data to obtain multi-channel reverberation noise data of the source noise data.

[0256] In order to obtain multi-channel reverberation-noised speech data, it is necessary to perform reverberation processing on the source noise data to obtain corresponding multi-channel reverberation noise data.

[0257] Among them, a setting method can be used to simulate the sound obtained by noise being reflected by objects in actual scenes, thereby realizing reverberation processing of source noise data.

[0258] Optionally, multi-channel reverberation noise data corresponding to the source noise data may be constructed based on a mirror image method.

[0259] Among them, the reflection of the source noise on the object can be simulated according to the object attribute information in the actual usage scenario to obtain noise data obtained by reflecting the source noise data.

[0260] For example, we can combine relevant object attribute information such as the size of the room in the actual usage scenario, the reflection coefficient of the wall, the room T60, the position parameters of the sound receiving device, and the position parameters of the sound source, randomly sample from the room parameter distribution, and use the room T60 to estimate the corresponding sound attenuation time.

[0261] The order of the mirror law is estimated according to the size of the room, and the impulse response of the source noise data is simulated and generated, and then the noise data obtained by the source noise data being reflected on the object is obtained according to the impulse response of the source noise data.

[0262] Furthermore, the noise data obtained by reflecting the source noise data may be mixed with the source noise data, and the noise data obtained after the mixing may be used as the multi-channel reverberation noise data obtained after the reverberation processing of the source noise data.

[0263] S604 : Mix the multi-channel reverberation speech data and the multi-channel reverberation noise data to obtain multi-channel reverberation noisy speech data, and generate training samples based on the multi-channel reverberation noisy speech data.

[0264] In the embodiment of the present disclosure, multi-channel reverberation speech data and multi-channel reverberation noise data may be mixed according to a set method to obtain mixed multi-channel reverberation and noisy speech data.

[0265] Before the multi-channel reverberation voice data and the multi-channel reverberation noise data are mixed, it is possible that the voice lengths of the multi-channel reverberation voice data and the multi-channel reverberation noise data to be mixed are different, so that effective mixing cannot be achieved. Therefore, it is necessary to sort out the voice lengths of the multi-channel reverberation voice data and the multi-channel reverberation noise data.

[0266] The speech length of the multi-channel reverberation noise data may be adjusted to be the same as the speech length of the multi-channel reverberation speech data.

[0267] Optionally, the speech length of the multi-channel reverberation speech data and the speech length of the multi-channel reverberation noise data can be obtained.

[0268] In a scenario where the speech length of the multi-channel reverberation speech data is less than the speech length of the multi-channel reverberation noise data, the speech length of the multi-channel reverberation noise data can be cut, and a segment with the same speech length as the multi-channel reverberation speech data is randomly cut off as the multi-channel reverberation noise data mixed with the multi-channel reverberation speech data.

[0269] In a scenario where the speech length of the multi-channel reverberation speech data is greater than the speech length of the multi-channel reverberation noise data, the multi-channel reverberation noise data can be copied and spliced at least once until the speech length of the spliced multi-channel reverberation noise data is the same as the speech length of the multi-channel reverberation speech data, and it is used as the multi-channel reverberation noise data mixed with the multi-channel reverberation speech data.

[0270] Further, the multi-channel reverberation noise data and the multi-channel reverberation speech data with the same speech length are mixed to obtain the mixed multi-channel reverberation noisy speech data.

[0271] Optionally, the multi-channel reverberation noise data and the multi-channel reverberation speech data with the same speech length can be mixed based on a set method.

[0272] Among them, the amplitudes of the multi-channel reverberation noise data and the multi-channel reverberation speech data with the same speech length can be obtained, and the signal-to-noise ratio of the multi-channel reverberation noise data and the multi-channel reverberation speech data with the same speech length can be randomly sampled based on a set decibel value range. The amplitudes of the multi-channel reverberation noise data and the multi-channel reverberation speech data with the same speech length are modulated according to the signal-to-noise ratio and added, and then the mixed multi-channel reverberation noisy speech data is obtained.

[0273] Further, samples for training the to-be-trained speech noise reduction model are generated according to the multi-channel reverberation noisy speech data, and training samples corresponding to the to-be-trained speech noise reduction model are generated according to multiple samples.

[0274] Optionally, the multi-channel reverberation noisy speech data can be labeled with label information, and the multi-channel reverberation noisy speech data after labeling the label is used as the training sample of the to-be-trained speech noise reduction model.

[0275] The training method of the noise reduction model proposed in the present disclosure performs reverberation processing on the source speech data in the source noisy speech data to obtain corresponding multi-channel reverberation speech data, performs reverberation processing on the source noise data in the source noisy speech data to obtain corresponding multi-channel reverberation noise data, further mixes the multi-channel reverberation speech data and the multi-channel reverberation noise data to generate corresponding multi-channel reverberation noisy speech data, and generates training samples based on the multi-channel reverberation noisy speech data. In the present disclosure, the corresponding multi-channel reverberation noisy speech data is obtained through reverberation processing, and the speech noise reduction model is trained using the multi-channel reverberation noisy speech data, thereby optimizing the training effect of the speech noise reduction model and making the trained speech noise reduction model have better applicability and practicality.

[0276] In order to obtain a better first speech noise reduction model and a second speech noise reduction model, after the model training is completed, the trained first speech noise reduction model and the second speech noise reduction model can be evaluated. Figure 7 Further understanding, Figure 7 FIG. 1 is a flow chart of a method for training a noise reduction model according to another embodiment of the present disclosure. Figure 7 As shown, the method includes:

[0277] S701 : Acquire evaluation reverberant and noisy speech data for evaluation based on multi-channel reverberant and noisy speech data.

[0278] In the embodiment of the present disclosure, the multi-channel reverberant and noisy speech data has corresponding evaluation reverberant and noisy speech data.

[0279] Among them, the evaluation reverberated and noisy speech data with a similar data distribution to the multi-channel reverberated and noisy speech data can be used as the verification reverberated and noisy speech data in the evaluation reverberated and noisy speech data, and the evaluation reverberated and noisy speech data with a different data distribution to the multi-channel reverberated and noisy speech data can be used as the test reverberated and noisy speech data in the evaluation reverberated and noisy speech data.

[0280] Optionally, the source noisy speech data can be segmented based on a set ratio, and multi-channel reverberated noisy speech data for model training can be obtained from the segmented speech data, and the evaluation reverberated noisy speech data for evaluation includes verification reverberated noisy speech data and test reverberated noisy speech data.

[0281] The split ratio can be set to 7:2:1, or other ratios, which are not limited here.

[0282] S702: Evaluate the first speech noise reduction model and / or the second speech noise reduction model based on the evaluated reverberant and noisy speech data.

[0283] In the embodiment of the present disclosure, the training effect evaluation of the first speech denoising model and the second speech denoising model can be achieved based on the verification reverberant and noisy speech data and the test reverberant and noisy speech data included in the evaluation reverberant and noisy speech data.

[0284] Among them, the evaluated reverberant noisy speech data can be input into the first speech noise reduction model and / or the second speech noise reduction model to obtain evaluation parameters of the first speech noise reduction model and / or the second speech noise reduction model, and the first speech noise reduction model and / or the second speech noise reduction model can be evaluated based on the evaluation parameters.

[0285] It should be noted that during the model evaluation process, there may be a situation where the evaluation effect does not meet the set standards. In this scenario, the first speech noise reduction model and / or the second speech noise reduction model can be re-trained based on the set multi-channel reverberation and noisy speech data until the evaluation effect of the first speech noise reduction model and / or the second speech noise reduction model obtained after retraining meets the set standards.

[0286] In some implementations, a loss function can be used as a corresponding evaluation parameter, wherein the verification reverberated and noisy speech data can be input into the first speech noise reduction model and the second speech noise reduction model, noise-free and reverberation-free speech data can be obtained from the verification reverberated and noisy speech data, and a loss function can be obtained between the denoised speech output by the first speech noise reduction model and the second speech noise reduction model and the noise-free and reverberation-free speech data in the verification reverberated and noisy speech data.

[0287] Furthermore, the loss function is compared with the loss functions of the last N rounds in the model training process, thereby evaluating the first speech denoising model and the second speech denoising model.

[0288] like Figure 8 As shown, Figure 8 This is the loss function curve under different model parameter configurations corresponding to the first speech noise reduction model.

[0289] Figure 8 It is divided into two rows from top to bottom, among which, Figure 8 The first row of graphs is the loss function curve of the first speech denoising model on multi-channel reverberation and noisy speech data. Figure 8 The second row of figures is the loss function curve of the first speech denoising model on the verification reverberant noisy speech data.

[0290] Figure 8 The first row of figures from left to right are the loss curves corresponding to the short-time objective intelligibility loss, the loss curve corresponding to the weighted total loss, and the loss curve corresponding to the scale-invariant signal-to-noise ratio loss. Figure 8The second row of figures shows, from left to right, the loss curve corresponding to the short-time objective intelligibility loss, the loss curve corresponding to the weighted total loss, and the loss curve corresponding to the scale-invariant signal-to-noise ratio loss.

[0291] Depend on Figure 8 It can be seen that the first speech denoising model converges to relatively good performance indicators after 120K training samples, and the indicators on the verification of reverberant and noisy speech data can meet the set standards. Therefore, the first speech denoising model has good generalization in the denoising processing of multi-channel reverberant and noisy speech data.

[0292] like Figure 9 As shown, Figure 9 This is the loss function curve under different model parameter configurations corresponding to the second speech noise reduction model.

[0293] Figure 9 It is divided into two rows from top to bottom, among which, Figure 9 The first row of graphs is the loss function curve of the second speech denoising model on multi-channel reverberation and noisy speech data. Figure 9 The second row of figures is the loss function curve of the second speech denoising model on the verification reverberant noisy speech data.

[0294] Figure 9 The first row of figures from left to right are the loss curves corresponding to the distillation loss, the loss curve corresponding to the weighted total loss, and the loss curve corresponding to the noise reduction loss. Figure 9 The second row of figures are, from left to right, the loss curves corresponding to the distillation loss, the loss curve corresponding to the weighted total loss, and the loss curve corresponding to the noise reduction loss.

[0295] Depend on Figure 9 It can be seen that the performance of the second speech denoising model is better than that of the first speech denoising model and the distillation loss converges. It can be determined that the intermediate features extracted by the feature extraction layer of the second speech denoising model are similar to the intermediate features extracted by the feature extraction layer of the first speech denoising model.

[0296] In some other implementations, the noise reduction effect of the second speech noise reduction model can be evaluated by testing reverberant and noisy speech data.

[0297] Optionally, the test reverberant and noisy speech data can be input into the second speech noise reduction model, and the scale-invariant signal-to-noise ratio (Si-SNR), short-time objective intelligibility (STOI), wideband speech quality evaluation index (WB-PESQ), narrowband speech quality evaluation index (NB-PESQ) and pure noise suppression index of the second speech noise reduction model on the test reverberant and noisy speech data are obtained.

[0298] Furthermore, the noise reduction effect of the second speech noise reduction model is scored, and its delay and real-time performance are measured.

[0299] In some other implementations, the first speech noise reduction model may be evaluated based on the mel-spectrogram as an evaluation parameter.

[0300] like Figure 10 As shown, Figure 10 The mel-spectrograms are the mel-spectrograms of the multi-channel reverberation and noisy speech data before and after the denoising process by the first speech denoising model, and the mel-spectrograms of the noise-free and reverberation-free speech data in the multi-channel reverberation and noisy speech data.

[0301] Figure 10 From left to right, it is divided into two columns, and each column is divided into three rows from top to bottom. The first row of the left column is the Mel spectrum diagram before noise reduction processing, the second row of the left column is the Mel spectrum diagram after noise reduction processing, and the third row of the left column is the Mel spectrum diagram of the speech data without noise and reverberation.

[0302] The noise in the multi-channel reverberant noisy speech data corresponding to the left column is broadband noise.

[0303] Figure 10 The first row of the right column is the Mel spectrum diagram before noise reduction processing, the second row of the right column is the Mel spectrum diagram after noise reduction processing, and the third row of the right column is the Mel spectrum diagram of the speech data without noise and reverberation.

[0304] The noise in the multi-channel reverberant noisy speech data corresponding to the right column is narrowband noise.

[0305] Depend on Figure 10 It can be seen that the first speech noise reduction model has a noise reduction effect that meets the set standard for broadband noise and narrowband noise, and the protection of the corresponding clean speech in the noise reduction process meets the set standard.

[0306] In other implementations, the first speech noise reduction model and the second speech noise reduction model can be compared with other existing algorithms, as shown in the following table, which includes four evaluation indicators: scale-invariant signal-to-noise ratio (Si-SNR), short-time objective intelligibility (STOI), speech quality evaluation index (PESQ) and model parameter quantity.

[0307] The FullSubNet and TCN algorithms include both their original and lightweight versions. The TCRN algorithm corresponds to the first speech noise reduction model in this disclosure and also has a corresponding lightweight version, which is the second speech noise reduction model corresponding to the first. The T-GSA, HiFi-GAN, and GeneralBeamformer algorithms only have their original versions.

[0308]

[0309] As can be seen from the above table, the last two rows in the table are the relevant performance indicators of the first speech noise reduction model and the second speech noise reduction model corresponding to the TCRN algorithm. They are the parameter values ​​of the first speech noise reduction model and the second speech noise reduction model in the embodiment of the present disclosure under the evaluation indicators implemented on the corresponding multi-channel reverberation and noisy speech data.

[0310] The first speech denoising model outperforms the other algorithms in the table in terms of scale-invariant signal-to-noise ratio, short-term objective intelligibility, and speech quality evaluation. The second speech denoising model, obtained after model distillation, shows a certain degree of improvement in performance compared to the first speech denoising model. This shows that model distillation for the first speech denoising model is effective and its impact on model performance meets the set standards, effectively demonstrating that the first and second speech denoising models have the robustness to meet the set standards.

[0311] As can be seen from the above table, the number of parameters of the lightweight TCN algorithm is slightly lower than that of the TCRN algorithm, but the performance is much different from that of the TCRN algorithm. Therefore, it can be determined that the first speech noise reduction model and the second speech noise reduction model in the embodiment of the present disclosure achieve an effective trade-off between performance and parameter quantity.

[0312] From the storage space dimension, the storage space occupied by the second speech noise reduction model is reduced to 0.05 times the storage space occupied by the first speech noise reduction model, and the compression ratio of the model storage space meets the set mark. Among them, the real-time rate of the second speech noise reduction model on the processor is significantly improved and is less than the set value of 0.5. In the scenario where the speech frame length of the input multi-channel reverberation and noisy speech data is 100 milliseconds and the real-time rate corresponding to the first speech noise reduction model processing the multi-channel reverberation and noisy speech data is less than 0.5, the noise reduction delay is less than the set value of 100 milliseconds.

[0313] The noise reduction model training method proposed in this disclosure evaluates the first and / or second speech noise reduction models using evaluation reverberant and noisy speech data corresponding to multi-channel reverberant and noisy speech data. This effectively demonstrates the training effectiveness of the first and second speech noise reduction models.

[0314] To better understand the above embodiments, Figure 11 , Figure 11 FIG. 1 is a flow chart of a method for training a noise reduction model according to another embodiment of the present disclosure. Figure 11 As shown, the method includes:

[0315] After obtaining the source noisy speech data, reverberation processing is performed on the source speech data in the source noisy speech data to obtain corresponding multi-channel reverberation speech data. Reverberation processing is also performed on the source noise data in the source noisy speech data to obtain corresponding multi-channel reverberation noise data. Furthermore, the multi-channel reverberation speech data and the multi-channel reverberation noise data are mixed to generate corresponding multi-channel reverberation noisy speech data. Furthermore, the multi-channel reverberation noisy speech data is segmented by frame to obtain single-frame multi-channel reverberation noisy speech data, and short-time Fourier transform is performed on the data to obtain a short-time spectrum corresponding to the single-frame multi-channel reverberation noisy speech data. The speech denoising model is trained using the short-time spectrum until training is completed, wherein the feature extraction layer of the speech denoising model extracts the time-domain noisy speech features and frequency-domain noisy speech features of the short-time spectrum, thereby obtaining the single-frame noisy speech features corresponding to the short-time spectrum. Furthermore, time-frequency masking is performed on the single-frame noisy speech features, thereby achieving time-frequency masking of the short-time spectrum and obtaining the corresponding spectrum mask. Based on the short-time spectrum and its corresponding spectral mask, a denoised short-time spectrum is obtained after the short-time spectrum is denoised. An inverse short-time Fourier transform is performed on the spectrum to obtain single-frame denoised speech data corresponding to the denoised short-time spectrum, which is used as the training output of the speech denoising model. Furthermore, a training loss function for the speech denoising model is obtained, and the model parameters of the speech denoising model are adjusted according to the training loss function until training is completed, thereby obtaining a trained first speech denoising model. A lightweight speech denoising model to be trained is constructed based on the first speech denoising model. The first speech denoising model is used as a teacher model, and the lightweight speech denoising model is trained using multi-channel reverberant and noisy speech data, so that the lightweight speech denoising model can learn from the first speech denoising model, thereby obtaining a lightweight second speech denoising model corresponding to the first speech denoising model. Furthermore, the first speech denoising model and / or the second speech denoising model are evaluated using evaluation reverberant and noisy speech data corresponding to the multi-channel reverberant and noisy speech data.

[0316] The training method of the noise reduction model provided in the present disclosure obtains corresponding multi-channel reverberation noisy speech data through reverberation processing, and trains the speech noise reduction model through the multi-channel reverberation noisy speech data, so that the human ear has good listening experience for the noise reduction speech output by the trained speech noise reduction model, improves the propagation accuracy of the information carried in the speech data, and reduces the influence degree of noise on the propagation of the information carried in the speech data. The multi-channel reverberation noisy speech data is framed, the speech noise reduction model is trained through the short-time spectrum corresponding to the single-frame multi-channel reverberation noisy speech data, the training method of the model is optimized, so that the trained speech noise reduction model realizes real-time noise reduction of noisy speech, and the lightweight second speech noise reduction model is obtained through model distillation of the first speech noise reduction model, so that the speech noise reduction model can be deployed on small terminals and micro terminals, and the applicability and practicability of the speech noise reduction model are improved.

[0317] The present disclosure also provides a speech noise reduction method, which can be combined with Figure 12 It is further understood that Figure 12 The flowchart of the speech noise reduction method of an embodiment of the present disclosure is shown in Figure 12 The method comprises the following steps:

[0318] S1201, obtaining multi-channel reverberation noisy speech to be de-noised, and performing short-time Fourier transform on the multi-channel reverberation noisy speech data to be de-noised by frame to generate a short-time spectrum to be de-noised of single-frame multi-channel reverberation noisy speech data to be de-noised.

[0319] In the present disclosure, the trained target speech noise reduction model can be used to de-noise the multi-channel reverberation noisy speech to be de-noised, wherein the target speech noise reduction model has a set standard for the input multi-channel reverberation noisy speech data to be de-noised.

[0320] The multi-channel reverberation noisy speech to be de-noised can be processed, and the processed multi-channel reverberation noisy speech to be de-noised can be input into the trained target speech noise reduction model.

[0321] Optionally, the multi-channel reverberation noisy speech data to be de-noised can be cut based on the time dimension, and the multi-channel reverberation noisy speech data to be de-noised can be cut and divided by frame to generate corresponding single-frame multi-channel reverberation noisy speech data to be de-noised.

[0322] Further, the single-frame multi-channel reverberation noisy speech data to be de-noised is subjected to short-time Fourier transform, and based on the result of the short-time Fourier transform, a short-time spectrum to be de-noised corresponding to the single-frame multi-channel reverberation noisy speech data to be de-noised is obtained.

[0323] S1202 , based on the trained target speech noise reduction model, perform noise reduction processing on the short-time spectrum to be noise reduced frame by frame to obtain the target speech data after noise reduction of the multi-channel reverberant noisy speech data to be noise reduced output by the target speech noise reduction model.

[0324] Among them, the target speech noise reduction model is Figures 1 to 11 The denoising model is trained using the training method shown.

[0325] In the embodiment of the present disclosure, the short-time spectrum to be denoised corresponding to a single frame of the multi-channel reverberant and noisy speech data to be denoised included in the multi-channel reverberant and noisy speech data to be denoised can be input into the trained target speech denoising model frame by frame.

[0326] Through the trained target speech noise reduction model, the short-time spectrum to be denoised is subjected to time-frequency masking to obtain the spectrum mask corresponding to the short-time spectrum to be denoised. Furthermore, based on the short-time spectrum to be denoised and its corresponding spectrum masking, the short-time spectrum corresponding to the short-time spectrum to be denoised after noise reduction processing is obtained.

[0327] The short-time spectrum is subjected to a short-time inverse Fourier transform to obtain a single-frame speech data after noise reduction corresponding to the single-frame multi-channel reverberation-noised speech data to be noise-reduced that is input into the target speech noise reduction model.

[0328] Furthermore, after all single-frame multi-channel reverberant and noisy speech data to be denoised included in the multi-channel reverberant and noisy speech data to be denoised are input into the target speech denoising model, target speech data after denoising the multi-channel reverberant and noisy speech data to be denoised is obtained.

[0329] It should be noted that the trained target speech noise reduction model in the embodiment of the present disclosure can be Figures 1 to 11 The denoising model is obtained by training using the training method proposed in the illustrated embodiment.

[0330] The speech denoising method proposed in the present disclosure performs short-time Fourier transform on the multi-channel reverberation-noisy speech data to be denoised frame by frame to generate a short-time spectrum of the multi-channel reverberation-noisy speech data to be denoised for a single frame. According to the trained target speech denoising model, the short-time spectrum to be denoised is subjected to denoising processing frame by frame to obtain the denoised target speech data of the multi-channel reverberation-noisy speech data to be denoised output by the target speech denoising model. In the present disclosure, by using the trained target speech denoising model, denoising processing is performed on the multi-channel reverberation-noisy speech data to be denoised frame by frame, thereby optimizing the immediacy of denoising the noisy speech, improving the accuracy of the propagation of the information carried in the speech data, reducing the influence of noise on the propagation of the information carried in the speech data, and optimizing the denoising method and denoising effect of the noisy speech.

[0331] Corresponding to the training method of the noise reduction model proposed in the above several embodiments, an embodiment of the present disclosure also proposes a training device of a noise reduction model. Since the training device of the noise reduction model proposed in the embodiment of the present disclosure corresponds to the training method of the noise reduction model proposed in the above several embodiments, the implementation manners of the above training method of the noise reduction model are also applicable to the training device of the noise reduction model proposed in the embodiment of the present disclosure, which will not be described in detail in the following embodiments.

[0332] Figure 13 The structural schematic diagram of the training device of the noise reduction model of an embodiment of the present disclosure is shown in FIG. 13, which comprises a reverberation module 131, a training module 132, a distillation module 133 and an evaluation module 134, wherein: Figure 13

[0333] The reverberation module 131 is configured to obtain source noisy speech data and perform reverberation processing on the source noisy speech data to generate corresponding training samples, wherein the training samples comprise a plurality of samples, and each sample comprises multi-channel reverberation noisy speech data.

[0334] The training module 132 is configured to train a to-be-trained speech noise reduction model based on the multi-channel reverberation noisy speech data included in the training samples to obtain a trained first speech noise reduction model.

[0335] In the embodiment of the present disclosure, the training module 132 is further configured to perform short-time Fourier transform on the multi-channel reverberation noisy speech data included in the training samples by frame to generate a short-time spectrum of single-frame multi-channel reverberation speech data, and train the to-be-trained speech noise reduction model according to the short-time spectrum to obtain the trained first speech noise reduction model.

[0336] In the embodiment of the present disclosure, the training module 132 is further configured to perform time-frequency masking on the short-time spectrum to obtain a spectral mask of the short-time spectrum, obtain a denoised short-time spectrum after denoising the short-time spectrum according to the spectral mask and the short-time spectrum, perform short-time inverse Fourier transform on the denoised short-time spectrum to generate speech data of the denoised short-time spectrum as single-frame denoised speech data of the training output of the speech noise reduction model, obtain a training loss function of the speech noise reduction model according to the single-frame denoised speech data, and adjust parameters of the speech noise reduction model according to the training loss function until the training is completed to obtain the trained first speech noise reduction model.

[0337] In the embodiment of the present disclosure, the training module 132 is further configured to extract single-frame noisy speech features of the short-time spectrum through a feature extraction layer of the speech noise reduction model, obtain a masking parameter of the single-frame noisy speech features, and perform time-frequency masking on the single-frame noisy speech features according to the masking parameter to obtain the spectral mask.

[0338] ​In the embodiment of the present disclosure, the training module 132 is further configured to: extract noisy speech features in the frequency domain and the time domain from the short-time spectrum through a feature extraction layer to obtain single-frame noisy speech features of the short-time spectrum.

[0339] In the embodiment of the present disclosure, the training module 132 is further used to: perform frequency domain feature extraction on the short-time spectrum based on the frequency domain causal hole convolution layer in the feature extraction layer to obtain frequency domain noisy speech features; perform time domain feature extraction on the short-time spectrum based on the time domain causal hole convolution layer in the feature extraction layer to obtain time domain noisy speech features; and obtain single-frame noisy speech features of the short-time spectrum based on the frequency domain noisy speech features and the time domain noisy speech features.

[0340] In the embodiment of the present disclosure, the training module 132 is further used to: extract the initial noisy speech features of the short-time spectrum in the causal hole convolution layer of the feature extraction layer; and obtain any noisy speech features of the short-time spectrum based on the initial noisy speech features and the sigmoid activation function.

[0341] In the embodiment of the present disclosure, the training module 132 is further used to: perform a first feature extraction on the short-time spectrum input to the causal convolutional layer, extract the first frame feature of the speech frame corresponding to the short-time spectrum, and the noisy speech feature corresponding to the first frame feature; obtain the first noisy speech feature of the short-time spectrum based on the first frame feature and the noisy speech feature corresponding to the first frame feature; discard part of the first noisy speech feature, calculate the remaining features in the first noisy speech feature after the discarding according to the ELU activation function, and use the calculated noisy speech feature as the initial noisy speech feature.

[0342] In the embodiment of the present disclosure, the training module 132 is further configured to: perform a second feature extraction on the initial noisy speech feature, extract a second frame feature of the speech frame corresponding to the initial noisy speech feature, and a noisy speech feature corresponding to the second frame feature; obtain a second noisy speech feature of the short-time spectrum based on the second frame feature and the noisy speech feature corresponding to the second frame feature; perform a reference calculation on the second noisy speech feature according to the sigmoid activation function, and use the calculated noisy speech feature as a third noisy speech feature; normalize the second noisy speech feature and the third noisy speech feature, and use the normalized noisy speech feature as any noisy speech feature of the short-time spectrum output by the causal convolution layer.

[0343] In the disclosed embodiment, when the causal convolution layer outputs frequency domain noisy speech features, the causal void convolution layer is a frequency domain causal void convolution layer; when the causal convolution layer outputs frequency domain noisy speech features, the causal void convolution layer is a time domain causal void convolution layer.

[0344] In the embodiment of the present disclosure, the training module 132 is further configured to: obtain single-frame speech data in the single-frame multi-channel reverberation noise speech data, and obtain a scale-invariant signal-to-noise ratio loss and a short-time objective intelligibility loss between the single-frame speech data and the single-frame noise-reduced speech data; and obtain a training loss function of the speech noise reduction model by weighted sum of the scale-invariant signal-to-noise ratio loss and the short-time objective intelligibility loss.

[0345] In the embodiment of the present disclosure, the training module 132 is further configured to: cache the single-frame noise speech feature as a historical single-frame noise speech feature to a cache unit, for waiting to be called next time.

[0346] In the embodiment of the present disclosure, the device further comprises a distillation module 133 configured to perform distillation processing on the first speech noise reduction model to obtain a lightweight second speech noise reduction model.

[0347] In the embodiment of the present disclosure, the distillation module 133 is further configured to: obtain a lightweight speech noise reduction model to be trained according to the first speech noise reduction model; obtain a distillation loss of the lightweight speech noise reduction model based on the first speech noise reduction model according to the multi-channel reverberation noise speech data included in the training sample; and perform parameter adjustment on the lightweight speech noise reduction model according to the distillation loss until the training is completed, to obtain the trained second speech noise reduction model.

[0348] In the embodiment of the present disclosure, the distillation module 133 is further configured to: obtain a lightweight extraction condition of a model structure of the first speech noise reduction model; and construct a to-be-trained model satisfying the lightweight extraction condition of the model structure as the lightweight speech noise reduction model.

[0349] In the embodiment of the present disclosure, the distillation module 133 is further configured to: obtain a feature extraction loss and a noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model according to the multi-channel reverberation noise speech data included in the training sample; and obtain the distillation loss of the lightweight speech noise reduction model based on the first speech noise reduction model by weighted sum of the feature extraction loss and the noise reduction loss.

[0350] In the embodiment of the present disclosure, the distillation module 133 is further configured to: obtain first noise speech features of the multi-channel reverberation noise speech data included in the training sample extracted by the first speech noise reduction model; obtain second noise speech features of the multi-channel reverberation noise speech data included in the training sample extracted by the lightweight speech noise reduction model; and obtain the feature extraction loss of the lightweight speech noise reduction model based on the first speech noise reduction model according to the first noise speech features and the second noise speech features.

[0351] In the embodiment of the present disclosure, the distillation module 133 is further used to: input the multi-channel reverberation and noisy speech data included in the training sample into the first speech noise reduction model to obtain the first noise-reduced speech output by the first speech noise reduction model; input the multi-channel reverberation and noisy speech data included in the training sample into the lightweight speech noise reduction model to obtain the second noise-reduced speech output by the lightweight speech noise reduction model; and obtain the noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model according to the first noise-reduced speech and the second noise-reduced speech.

[0352] In the embodiment of the present disclosure, the reverberation module 131 is further used to: obtain source speech data and source noise data in the source noisy speech data; perform reverberation processing on the source speech data to obtain multi-channel reverberation speech data of the source speech data; perform reverberation processing on the source noise data to obtain multi-channel reverberation noise data of the source noise data; mix the multi-channel reverberation speech data and the multi-channel reverberation noise data to obtain multi-channel reverberation noisy speech data, and generate the training sample based on the multi-channel reverberation noisy speech data.

[0353] In the embodiment of the present disclosure, the reverberation module 131 is further used to: obtain initial speech data and initial noise data; perform data cleaning and data enhancement on the initial speech data and the initial noise data, respectively, to obtain source speech data corresponding to the initial speech data and source noise data corresponding to the initial noise data; and splice the source speech data and the source noise data to generate the source noisy speech data.

[0354] In the embodiment of the present disclosure, the reverberation module 131 is further used to: adjust the speech length of the multi-channel reverberation noise data to be the same as the speech length of the multi-channel reverberation speech data; mix the multi-channel reverberation noise data and the multi-channel reverberation speech data with the same speech length to obtain mixed multi-channel reverberation noisy speech data.

[0355] In the embodiment of the present disclosure, the evaluation module 134 is used to: obtain evaluation reverberant and noisy speech data for evaluation based on the multi-channel reverberant and noisy speech data; and evaluate the first speech noise reduction model and / or the second speech noise reduction model based on the evaluation reverberant and noisy speech data.

[0356] In the embodiment of the present disclosure, the evaluation module 134 is further used to: obtain evaluation parameters of the first speech noise reduction model and / or the second speech noise reduction model, and evaluate the first speech noise reduction model and / or the second speech noise reduction model according to the evaluation parameters.

[0357] The training device of the noise reduction model proposed in the present invention obtains corresponding multi-channel reverberation and noisy speech data as samples through reverberation processing, and trains the speech noise reduction model through the multi-channel reverberation and noisy speech data, so that the human ear has a good hearing experience for the noise-reduced speech output by the trained speech noise reduction model, improves the transmission accuracy of the information carried in the speech data, and reduces the influence of noise on the transmission of the information carried in the speech data. The multi-channel reverberation and noisy speech data is framed, and the speech noise reduction model is trained through the short-time spectrum corresponding to the single-frame multi-channel reverberation and noisy speech data, thereby optimizing the training method of the model, so that the trained speech noise reduction model can achieve real-time noise reduction of noisy speech, and a lightweight second speech noise reduction model is obtained by model distillation of the first speech noise reduction model, so that the speech noise reduction model can be deployed on small terminals and micro terminals, thereby improving the applicability and practicality of the speech noise reduction model.

[0358] Corresponding to the speech noise reduction methods proposed in the above-mentioned embodiments, an embodiment of the present disclosure also proposes a speech noise reduction device. Since the speech noise reduction device proposed in the embodiment of the present disclosure corresponds to the speech noise reduction methods proposed in the above-mentioned embodiments, the implementation methods of the above-mentioned speech noise reduction methods are also applicable to the speech noise reduction device proposed in the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0359] Figure 14 FIG. 1 is a structural diagram of a speech noise reduction device according to an embodiment of the present disclosure. Figure 14 As shown, the speech noise reduction device 1400 includes an acquisition module 141 and a noise reduction module 142, wherein:

[0360] An acquisition module 141 is configured to acquire multi-channel reverberant and noisy speech data to be denoised, and perform short-time Fourier transform on the multi-channel reverberant and noisy speech data to be denoised frame by frame to generate a short-time spectrum of the multi-channel reverberant and noisy speech data to be denoised for a single frame;

[0361] The denoising module 142 is configured to perform denoising processing on the short-time spectrum to be denoised frame by frame based on the trained target speech denoising model to obtain denoised target speech data of the multi-channel reverberant noisy speech data to be denoised output by the target speech denoising model.

[0362] The present disclosure proposes a speech noise reduction device that performs short-time Fourier transform on multi-channel reverberation-noisy speech data to be denoised frame by frame to generate a short-time spectrum of a single frame of multi-channel reverberation-noisy speech data to be denoised, and performs noise reduction processing on the short-time spectrum of the to-be-denoised speech data frame by frame according to a trained target speech noise reduction model to obtain the target speech data after noise reduction of the multi-channel reverberation-noisy speech data to be denoised output by the target speech noise reduction model. In the present disclosure, by using a trained target speech noise reduction model, noise reduction processing is performed on the multi-channel reverberation-noisy speech data to be denoised frame by frame, thereby optimizing the immediacy of noise reduction of noisy speech, improving the accuracy of the transmission of information carried in the speech data, reducing the degree of influence of noise on the transmission of information carried in the speech data, and optimizing the noise reduction method and noise reduction effect of noisy speech.

[0363] To achieve the above embodiments, the present disclosure also provides an electronic device, a computer-readable storage medium, and a computer program product.

[0364] Figure 15 1 shows a schematic block diagram of an example electronic device 1500 that can be used to implement embodiments of the present disclosure. Figure 15 As shown, the device 1500 includes a memory 151, a processor 152, and a computer program stored in the memory 151 and executable on the processor 152. When the processor 152 executes the program instructions, the training method of the noise reduction model and / or the speech noise reduction method provided in the above embodiments are implemented.

[0365] After obtaining the source noisy speech data, reverberation processing is performed on the source speech data in the source noisy speech data to obtain corresponding multi-channel reverberation speech data. Reverberation processing is also performed on the source noise data in the source noisy speech data to obtain corresponding multi-channel reverberation noise data. Furthermore, the multi-channel reverberation speech data and the multi-channel reverberation noise data are mixed to generate corresponding multi-channel reverberation noisy speech data. Furthermore, the multi-channel reverberation noisy speech data is segmented by frame to obtain single-frame multi-channel reverberation noisy speech data, and short-time Fourier transform is performed on the data to obtain a short-time spectrum corresponding to the single-frame multi-channel reverberation noisy speech data. The speech denoising model is trained using the short-time spectrum until training is completed, wherein the feature extraction layer of the speech denoising model extracts the time-domain noisy speech features and frequency-domain noisy speech features of the short-time spectrum, thereby obtaining the single-frame noisy speech features corresponding to the short-time spectrum. Furthermore, time-frequency masking is performed on the single-frame noisy speech features, thereby achieving time-frequency masking of the short-time spectrum and obtaining the corresponding spectrum mask. Based on the short-time spectrum and its corresponding spectral mask, a denoised short-time spectrum is obtained after the short-time spectrum is denoised. An inverse short-time Fourier transform is performed on the spectrum to obtain single-frame denoised speech data corresponding to the denoised short-time spectrum, which is used as the training output of the speech denoising model. Furthermore, a training loss function for the speech denoising model is obtained, and the model parameters of the speech denoising model are adjusted according to the training loss function until training is completed, thereby obtaining a trained first speech denoising model. A lightweight speech denoising model to be trained is constructed based on the first speech denoising model. The first speech denoising model is used as a teacher model, and the lightweight speech denoising model is trained using multi-channel reverberant and noisy speech data, so that the lightweight speech denoising model can learn from the first speech denoising model, thereby obtaining a lightweight second speech denoising model corresponding to the first speech denoising model. Furthermore, the first speech denoising model and / or the second speech denoising model are evaluated using evaluation reverberant and noisy speech data corresponding to the multi-channel reverberant and noisy speech data. In the present disclosure, corresponding multi-channel reverberation and noisy speech data is obtained through reverberation processing, and a speech noise reduction model is trained through the multi-channel reverberation and noisy speech data, so that the human ear has a good hearing experience for the noise-reduced speech output by the trained speech noise reduction model, the transmission accuracy of the information carried in the speech data is improved, and the influence of noise on the transmission of the information carried in the speech data is reduced. The multi-channel reverberation and noisy speech data is framed, and the speech noise reduction model is trained through the short-time spectrum corresponding to the single-frame multi-channel reverberation and noisy speech data, the training method of the model is optimized, so that the trained speech noise reduction model can achieve real-time noise reduction of noisy speech, and a lightweight second speech noise reduction model is obtained through model distillation of the first speech noise reduction model, so that the speech noise reduction model can be deployed on small terminals and micro terminals, thereby improving the applicability and practicality of the speech noise reduction model.

[0366] The short-time Fourier transform is performed on the multi-channel reverberation noisy speech data to be denoised frame by frame to generate a short-time spectrum to be denoised of the single frame of multi-channel reverberation noisy speech data to be denoised, and the short-time spectrum to be denoised is processed frame by frame according to the trained target speech denoising model to obtain the denoised target speech data of the multi-channel reverberation noisy speech data to be denoised output by the target speech denoising model. In the present disclosure, the multi-channel reverberation noisy speech data to be denoised is processed frame by frame according to the trained target speech denoising model, the instantaneity of noisy speech denoising is optimized, the propagation accuracy of information carried in the speech data is improved, the influence of noise on the propagation of information carried in the speech data is reduced, and the denoising method and effect of noisy speech are optimized.

[0367] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0368] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0369] Program code for carrying out operations of the methods described herein can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a remote machine or entirely on a remote machine or server.

[0370] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0371] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0372] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or grid browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by digital data communication (e.g., a communication grid) in any form or medium. Examples of communication grids include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain grid.

[0373] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication grid. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service system that addresses the management difficulties and poor business scalability of traditional physical hosts and VPS services ("Virtual Private Server," or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0374] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and features of different embodiments or examples, unless they are mutually inconsistent.

[0375] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0376] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0377] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0378] It should be understood that various parts of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0379] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0380] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0381] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. A person of ordinary skill in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

[0382] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0383] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training a denoising model, characterized in that: The method comprises: Acquire source speech data and source noise data from source noisy speech data; perform reverberation processing on the source speech data to obtain multi-channel reverberation speech data of the source speech data; perform reverberation processing on the source noise data to obtain multi-channel reverberation noise data of the source noise data; mix the multi-channel reverberation speech data and the multi-channel reverberation noise data to obtain multi-channel reverberation-noisy speech data; generate corresponding training samples based on the multi-channel reverberation-noisy speech data, wherein the training samples include a plurality of samples, each of the samples includes multi-channel reverberation-noisy speech data; Short-time Fourier transform is performed on the multi-channel reverberation-noisy speech data frame by frame to generate a short-time spectrum of a single-frame multi-channel reverberation speech data; and a speech noise reduction model to be trained is trained based on the short-time spectrum of the single-frame multi-channel reverberation speech data to obtain a trained first speech noise reduction model.

2. The method according to claim 1, characterized in that The step of training the speech noise reduction model to be trained according to the short-time spectrum to obtain a trained first speech noise reduction model includes: Performing time-frequency masking on the short-time spectrum to obtain a spectrum mask of the short-time spectrum; Acquire a denoised short-time spectrum after denoising the short-time spectrum according to the spectrum mask and the short-time spectrum; Performing a short-time inverse Fourier transform on the denoised short-time spectrum to generate speech data of the denoised short-time spectrum as single-frame denoised speech data output by training the speech denoising model; A training loss function of the speech noise reduction model is obtained according to the single-frame denoised speech data, and parameters of the speech noise reduction model are adjusted according to the training loss function until the training is completed, thereby obtaining the trained first speech noise reduction model.

3. The method according to claim 2, characterized in that The performing time-frequency masking on the short-time spectrum to obtain a spectrum mask of the short-time spectrum includes: Extracting the single-frame noisy speech features of the short-time spectrum through the feature extraction layer of the speech noise reduction model; Obtaining a masking parameter of the single-frame noisy speech feature, and performing time-frequency masking on the single-frame noisy speech feature according to the masking parameter to obtain the spectrum mask.

4. The method according to claim 3, characterized in that The extracting the single-frame noisy speech feature of the short-time spectrum by the speech noise reduction model includes: The feature extraction layer performs frequency domain and time domain noisy speech feature extraction on the short-time spectrum to obtain the single-frame noisy speech feature of the short-time spectrum.

5. The method according to claim 4, characterized in that The step of extracting noisy speech features in the frequency domain and the time domain from the short-time spectrum through the feature extraction layer to obtain the single-frame noisy speech features of the short-time spectrum includes: Performing frequency domain feature extraction on the short-time spectrum based on the frequency domain causal dilated convolution layer in the feature extraction layer to obtain frequency domain noisy speech features; Performing time domain feature extraction on the short-time spectrum based on the time domain causal dilated convolution layer in the feature extraction layer to obtain time domain noisy speech features; The single-frame noisy speech feature of the short-time spectrum is obtained according to the frequency-domain noisy speech feature and the time-domain noisy speech feature.

6. The method according to claim 5, characterized in that Any of the frequency-domain noisy speech features and the time-domain noisy speech features includes: In the causal dilated convolution layer of the feature extraction layer, extracting the initial noisy speech features of the short-time spectrum; According to the initial noisy speech feature and the sigmoid activation function, any noisy speech feature of the short-time spectrum is obtained.

7. The method according to claim 6, characterized in that Extracting the initial noisy speech features of the short-time spectrum in the causal convolution layer of the feature extraction layer includes: Performing a first feature extraction on the short-time spectrum input to the causal convolutional layer, extracting a first frame feature of a speech frame corresponding to the short-time spectrum and a noisy speech feature corresponding to the first frame feature; Acquire a first noisy speech feature of the short-time spectrum according to the first frame feature and a noisy speech feature corresponding to the first frame feature; Partial feature discarding is performed on the first noisy speech feature, remaining features in the discarded first noisy speech feature are calculated according to the ELU activation function, and the calculated noisy speech feature is used as the initial noisy speech feature.

8. The method according to claim 7, characterized in that The step of obtaining any one of the noisy speech features of the short-time spectrum extracted by the causal convolution layer according to the initial noisy speech feature and the sigmoid activation function includes: Performing a second feature extraction on the initial noisy speech feature to extract a second frame feature of the speech frame corresponding to the initial noisy speech feature and a noisy speech feature corresponding to the second frame feature; Acquire a second noisy speech feature of the short-time spectrum according to the second frame feature and the noisy speech feature corresponding to the second frame feature; performing a reference calculation on the second noisy speech feature according to the sigmoid activation function, and using the calculated noisy speech feature as the third noisy speech feature; The second noisy speech feature and the third noisy speech feature are normalized, and the noisy speech feature obtained by normalization is used as any one of the noisy speech features of the short-time spectrum output by the causal convolution layer.

9. The method according to any one of claims 6 to 8, characterized in that: When the causal convolution layer outputs the frequency domain noisy speech feature, the causal dilated convolution layer is the frequency domain causal dilated convolution layer; When the causal convolution layer outputs the frequency domain noisy speech feature, the causal void convolution layer is the time domain causal void convolution layer.

10. The method according to claim 2, characterized in that The method of obtaining a training loss function of the speech noise reduction model according to the single-frame noise reduction speech data, and adjusting parameters of the speech noise reduction model according to the training loss function until the training is completed to obtain the trained first speech noise reduction model includes: Acquire single-frame speech data from the multi-channel reverberated noisy speech data, and acquire a scale-invariant signal-to-noise ratio loss and a short-time objective intelligibility loss between the single-frame speech data and the single-frame denoised speech data; A weighted sum is performed on the scale-invariant signal-to-noise ratio loss and the short-time objective intelligibility loss to obtain the training loss function of the speech noise reduction model.

11. The method according to claim 3, characterized in that After obtaining the single-frame noisy speech feature of the short-time spectrum, the method includes: The single-frame noisy speech feature is cached in a cache unit as a historical single-frame noisy speech feature, waiting to be called next time.

12. The method according to claim 1, characterized in that The method further comprises: The first speech noise reduction model is subjected to distillation processing to obtain a lightweight second speech noise reduction model.

13. The method according to claim 12, characterized in that The distilling process of the first speech noise reduction model to obtain a lightweight second speech noise reduction model includes: According to the first speech noise reduction model, obtaining a lightweight speech noise reduction model to be trained; Obtaining, according to the multi-channel reverberant noisy speech data included in the training sample, a distillation loss of the lightweight speech denoising model based on the first speech denoising model; Parameters of the lightweight speech denoising model are adjusted according to the distillation loss until training is completed, thereby obtaining a trained second speech denoising model.

14. The method according to claim 13, characterized in that The step of obtaining a lightweight speech noise reduction model to be trained according to the first speech noise reduction model includes: Obtaining a lightweight extraction condition of the model structure of the first speech noise reduction model; A model to be trained whose model structure meets the lightweight extraction conditions is constructed as the lightweight speech noise reduction model.

15. The method according to claim 14, characterized in that The obtaining, according to the multi-channel reverberant noisy speech data included in the training sample, a distillation loss of the lightweight speech denoising model based on the first speech denoising model comprises: According to the multi-channel reverberant noisy speech data included in the training sample, obtaining a feature extraction loss and a noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model; A weighted sum is performed on the feature extraction loss and the noise reduction loss to obtain the distillation loss of the lightweight speech noise reduction model based on the first speech noise reduction model.

16. The method according to claim 15, characterized in that The process of obtaining the feature extraction loss includes: Acquire a first noisy speech feature of the multi-channel reverberant noisy speech data included in the training sample extracted by the first speech noise reduction model; Acquire a second noisy speech feature of the multi-channel reverberant noisy speech data included in the training sample extracted by the lightweight speech noise reduction model; The feature extraction loss of the lightweight speech noise reduction model based on the first speech noise reduction model is obtained according to the first noisy speech feature and the second noisy speech feature.

17. The method according to claim 15, characterized in that The process of obtaining the noise reduction loss includes: Inputting the multi-channel reverberant noisy speech data included in the training sample into the first speech noise reduction model to obtain a first noise-reduced speech output by the first speech noise reduction model; Inputting the multi-channel reverberant noisy speech data included in the training sample into the lightweight speech noise reduction model to obtain a second noise-reduced speech output by the lightweight speech noise reduction model; According to the first noise reduction speech and the second noise reduction speech, the noise reduction loss of the lightweight speech noise reduction model based on the first speech noise reduction model is obtained.

18. The method according to claim 1, wherein Before obtaining the source speech data and the source noise data in the source noisy speech data, the method includes: Acquire initial speech data and initial noise data; Performing data cleaning and data enhancement on the initial speech data and the initial noise data, respectively, to obtain source speech data corresponding to the initial speech data and source noise data corresponding to the initial noise data; The source speech data and the source noise data are concatenated to generate the source noisy speech data.

19. The method according to claim 1, wherein Mixing the multi-channel reverberation speech data and the multi-channel reverberation noise data to obtain the multi-channel reverberation noisy speech data, comprising: Adjusting the speech length of the multi-channel reverberation noise data to be the same as the speech length of the multi-channel reverberation speech data; The multi-channel reverberation noise data and the multi-channel reverberation speech data having the same speech length are mixed to obtain the mixed multi-channel reverberation noisy speech data.

20. The method according to any one of claims 12 to 19, characterized in that: The method further comprises: Acquiring evaluation reverberant and noisy speech data for evaluation based on the multi-channel reverberant and noisy speech data; The first speech noise reduction model and / or the second speech noise reduction model are evaluated according to the evaluated reverberant and noisy speech data.

21. The method according to claim 20, characterized in that The evaluating the first speech noise reduction model and / or the second speech noise reduction model according to the evaluated reverberant and noisy speech data includes: Acquire evaluation parameters of the first speech noise reduction model and / or the second speech noise reduction model, and evaluate the first speech noise reduction model and / or the second speech noise reduction model according to the evaluation parameters.

22. A method for reducing speech noise, characterized in that: The method comprises: Acquire multi-channel reverberant and noisy speech data to be denoised, and perform short-time Fourier transform on the multi-channel reverberant and noisy speech data to be denoised frame by frame to generate a short-time spectrum to be denoised of a single frame of the multi-channel reverberant and noisy speech data to be denoised; Based on the trained target speech noise reduction model, performing noise reduction processing on the short-time spectrum to be denoised frame by frame to obtain the denoised target speech data of the multi-channel reverberant noisy speech data to be denoised output by the target speech noise reduction model; The target speech noise reduction model is obtained by training using the noise reduction model training method described in any one of claims 1 to 20.

23. A training device for a noise reduction model, characterized in that: The device comprises: A reverberation module is configured to obtain source speech data and source noise data from source noisy speech data; perform reverberation processing on the source speech data to obtain multi-channel reverberation speech data of the source speech data; perform reverberation processing on the source noise data to obtain multi-channel reverberation noise data of the source noise data; mix the multi-channel reverberation speech data and the multi-channel reverberation noise data to obtain multi-channel reverberation-noisy speech data; and generate corresponding training samples based on the multi-channel reverberation-noisy speech data, wherein the training samples include a plurality of samples, each of which includes multi-channel reverberation-noisy speech data. The training module is used to perform short-time Fourier transform on the multi-channel reverberation noisy speech data included in the training sample frame by frame to generate a short-time spectrum of a single-frame multi-channel reverberation speech data; and train the speech denoising model to be trained based on the short-time spectrum of the single-frame multi-channel reverberation speech data to obtain a trained first speech denoising model.

24. A speech noise reduction device, characterized in that: The device comprises: An acquisition module is used to acquire multi-channel reverberant and noisy speech data to be denoised, and perform short-time Fourier transform on the multi-channel reverberant and noisy speech data to be denoised frame by frame to generate a short-time spectrum to be denoised of a single frame of the multi-channel reverberant and noisy speech data to be denoised; A noise reduction module is configured to perform noise reduction processing on the short-time spectrum to be denoised frame by frame based on a trained target speech noise reduction model, so as to obtain target speech data after noise reduction of the multi-channel reverberant noisy speech data to be denoised output by the target speech noise reduction model; The target speech noise reduction model is obtained by training using the noise reduction model training method described in any one of claims 1 to 20.

25. An electronic device, characterized in that: include: processor; a memory for storing executable instructions for the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 22. 26 . A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to claim 1 .

Images