Voice denoising network training method and device, electronic equipment and storage medium

Abstract

The application provides a speech noise reduction network training method and device, electronic equipment and computer readable storage medium, comprising: performing short-time Fourier transform on sample speech data in a sample data set to obtain sample time-frequency domain features; the sample speech data includes noise speech data and clean speech data, and the sample time-frequency domain features include noise time-frequency domain features and clean time-frequency domain features; calculating the noise time-frequency domain features through a neural network model to obtain predicted time-frequency domain features; evaluating the difference between the predicted time-frequency domain features and the clean time-frequency domain features through a loss function to obtain a function value, and determining whether the function value is less than a preset loss threshold; wherein the loss function is switched periodically during the training process; if yes, it is determined that the neural network model converges, and a speech noise reduction network is obtained. According to the application, the speech noise reduction network considering noise reduction and speech fidelity effect is trained without increasing model parameters.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a training method and apparatus for a speech denoising network, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With advancements in technology, speech denoising algorithms implemented using neural network models have become widely used. However, when the neural network model is small, a speech denoising network trained directly end-to-end cannot simultaneously achieve good results in terms of both speech distortion and noise reduction. In other words, in this situation, if the speech denoising network aims to achieve minimal speech distortion after denoising, the denoising effect will be poor; conversely, if the speech denoising network aims to achieve a large amount of noise reduction, the speech distortion will be severe.

[0003] This problem can generally be solved by increasing the model parameters, mainly by increasing the depth and width of the neural network model. This method increases the size of the neural network model, leading to increased computational load and memory usage. For devices with limited hardware resources, this method can negatively impact the operation of the speech denoising process.

[0004] In view of this, there is an urgent need for a solution that can train a speech denoising network without increasing the model parameters. Summary of the Invention

[0005] The purpose of this application is to provide a training method and apparatus, electronic device, and computer-readable storage medium for a speech denoising network, which can be trained to obtain a speech denoising network that balances noise reduction and speech fidelity without increasing model parameters.

[0006] On the one hand, this application provides a method for training a speech denoising network, including:

[0007] A short-time Fourier transform is performed on the sample speech data in the sample dataset to obtain the sample time-frequency domain features; wherein, the sample speech data includes noisy speech data and clean speech data corresponding to the noisy speech data, and the sample time-frequency domain features include the noise time-frequency domain features obtained by transforming the noisy speech data and the clean time-frequency domain features obtained by transforming the clean speech data.

[0008] The predicted time-frequency domain features are obtained by calculating the time-frequency domain features of the noise using a neural network model.

[0009] The difference between the predicted time-frequency domain features and the clean time-frequency domain features is evaluated by a loss function to obtain a function value, and the model parameters of the neural network model are adjusted based on the function value.

[0010] Determine whether the function value is less than a preset loss threshold; wherein, the loss function is switched periodically during the training process.

[0011] If so, the neural network model is determined to have converged, thus obtaining the speech denoising network.

[0012] In one embodiment, the method further includes:

[0013] If the function value is not less than the loss threshold, return to the step of calculating the noise time-frequency domain features through a neural network model to obtain the predicted time-frequency domain features.

[0014] In one embodiment, before performing a short-time Fourier transform on the sample speech data in the sample dataset to obtain the sample time-frequency domain features, the method further includes:

[0015] Multiple clean speech data are acquired, and corresponding noisy speech data is generated for each clean speech data; wherein, the noisy speech data includes clean speech data and noise data;

[0016] Each clean speech data point is used as a sample label for its corresponding noisy speech data.

[0017] The sample dataset is constructed based on multiple noisy speech data carrying sample labels.

[0018] In one embodiment, the neural network model includes a first fully connected layer, a first feature processing module, a second feature processing module, a third feature processing module, and a second fully connected layer; wherein the first fully connected layer is connected to the first feature processing module, the first fully connected layer, the first feature processing module, and the second feature processing module are residually connected, the first fully connected layer, the first feature processing module, the second feature processing module, and the third feature processing module are residually connected, and the third feature processing module is connected to the second fully connected layer.

[0019] In one embodiment, the first feature processing module, the second feature processing module, and the third feature processing module are recurrent neural networks.

[0020] In one embodiment, the loss function is switched in stages during the training process, including:

[0021] During training, MAE loss, MSE loss, and fourth power difference loss were selected in sequence.

[0022] In one embodiment, after obtaining the speech denoising network, the method further includes:

[0023] The speech data to be processed is subjected to a short-time Fourier transform to obtain the time-frequency domain features to be processed.

[0024] The speech denoising network is used to calculate the time-frequency domain features to be processed, and the denoised time-frequency domain features are obtained.

[0025] The denoised time-frequency domain features are subjected to inverse Fourier transform to obtain the denoised speech data.

[0026] On the other hand, this application provides a training apparatus for a speech denoising network, comprising:

[0027] The transformation module is used to perform short-time Fourier transform on the sample speech data in the sample dataset to obtain the sample time-frequency domain features; wherein, the sample speech data includes noisy speech data and clean speech data corresponding to the noisy speech data, and the sample time-frequency domain features include the noise time-frequency domain features obtained by transforming the noisy speech data, and the clean time-frequency domain features obtained by transforming the clean speech data.

[0028] The calculation module is used to calculate the time-frequency domain features of the noise using a neural network model to obtain the predicted time-frequency domain features;

[0029] The adjustment module is used to evaluate the difference between the predicted time-frequency domain features and the clean time-frequency domain features through a loss function, obtain a function value, and adjust the model parameters of the neural network model based on the function value;

[0030] The judgment module is used to determine whether the function value is less than a preset loss threshold; wherein, the loss function is switched periodically during the training process.

[0031] The determination module is used to determine if the neural network model has converged, thereby obtaining the speech denoising network.

[0032] Furthermore, this application provides an electronic device, the electronic device comprising:

[0033] processor;

[0034] Memory used to store processor-executable instructions;

[0035] The processor is configured to execute the training method of the above-described speech denoising network.

[0036] In addition, this application provides a computer-readable storage medium storing a computer program that can be executed by a processor to complete the training method of the above-described speech denoising network.

[0037] The proposed solution first converts speech data into time-frequency domain features during training, enabling the neural network model to process language data as time-frequency domain features more comprehensively. Through multi-stage loss optimization, the loss function is switched during training to optimize features with different numerical ranges, allowing the trained speech denoising network to reduce distortion while achieving sufficiently good denoising results. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below.

[0039] Figure 1 A schematic diagram illustrating an application scenario of the training method for a speech denoising network provided in an embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0041] Figure 3 A schematic flowchart illustrating a training method for a speech denoising network provided in an embodiment of this application;

[0042] Figure 4 A schematic diagram of the training process of a speech denoising network provided in an embodiment of this application;

[0043] Figure 5 A schematic diagram of the architecture of a neural network model provided in an embodiment of this application;

[0044] Figure 6 This is a block diagram of a training apparatus for a speech denoising network provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0046] Similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] Figure 1 This is a schematic diagram illustrating an application scenario of the training method for the speech denoising network provided in this application embodiment. For example... Figure 1As shown, the application scenario includes a client 20 and a server 30. The client 20 can be a user terminal such as a host, mobile phone, or tablet computer, used to initiate training requests to the server 30. The server 30 can be a server, server cluster, or cloud computing center, which can respond to the training request and execute the training method of the speech denoising network.

[0048] like Figure 2 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12. Figure 2 Taking a processor 11 as an example, the processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 to enable the electronic device 1 to perform all or part of the processes of the methods in the embodiments described below. In one embodiment, the electronic device 1 may be the server 30 described above, used to execute the training method of the speech denoising network.

[0049] The memory 12 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0050] This application also provides a computer-readable storage medium storing a computer program that can be executed by a processor 11 to complete the training method for the speech denoising network provided in this application.

[0051] See Figure 3 This is a flowchart illustrating a training method for a speech denoising network provided in an embodiment of this application. Figure 3 As shown, the method may include steps 310-340.

[0052] Step 310: Perform a short-time Fourier transform on the sample speech data in the sample dataset to obtain the sample time-frequency domain features; wherein, the sample speech data includes noisy speech data and clean speech data corresponding to the noisy speech data, and the sample time-frequency domain features include the noise time-frequency domain features obtained by transforming the noisy speech data, and the clean time-frequency domain features obtained by transforming the clean speech data.

[0053] Each sample speech data in the sample dataset includes one noisy speech data entry and one clean speech data entry, where the clean speech data is the denoised speech data. In one embodiment, the speech data can be a 16K sampled WAV file. Depending on the application requirements, the speech data can also be other data file formats.

[0054] The server can perform a short-time Fourier transform (STFT) on each sample speech data to obtain the sample's time-frequency domain features. In this case, the sample speech data is transformed from time-domain features to time-frequency domain features, making it easier for subsequent processing.

[0055] Step 320: Calculate the time-frequency domain features of the noise using a neural network model to obtain the predicted time-frequency domain features.

[0056] After obtaining the time-frequency domain features of the samples, the server can input the noise time-frequency domain features from the sample time-frequency domain features into the neural network model. The neural network model then performs forward computation to output the predicted time-frequency domain features. Here, the predicted time-frequency domain features include a frequency domain mask used to block noise information, representing the predicted noise reduction result of the noise time-frequency domain features.

[0057] Step 330: Evaluate the difference between the predicted time-frequency domain features and the clean time-frequency domain features using the loss function, obtain the function value, and determine whether the function value is less than the preset loss threshold; wherein, the loss function is switched in stages during the training process.

[0058] Step 340: If yes, confirm that the neural network model has converged to obtain the speech denoising network.

[0059] After obtaining the predicted time-frequency domain features, for the same noisy time-frequency domain features, the server can evaluate the difference between the predicted and clean time-frequency domain features using a loss function, thus obtaining the function value of the loss function. After obtaining the function value, the model parameters of the neural network model can be adjusted based on this value, thereby updating the neural network model.

[0060] The server can determine whether the function value is less than a preset loss threshold, which can be pre-configured based on experience.

[0061] In one scenario, if the function value is not less than the loss threshold, it indicates that the neural network model has not converged. The server can return to step 320, recalculate the noise time-frequency domain features using the updated neural network model, and continue to execute step 330 after obtaining the new predicted time-frequency domain features.

[0062] During iterative training, the server can switch loss functions periodically. For example, multiple loss functions can be configured with corresponding iteration counts and switching order. For instance, the first loss function might be used 100 times during training, the second loss function 100 times, and the third loss function 100 times, with the loss function switching every 100 iterations. Alternatively, multiple loss functions can be configured with a corresponding switching order; during training, when the function value calculated by any one loss function tends to stabilize, the next loss function can be selected.

[0063] In one embodiment, when switching loss functions in stages, MAE (Mean Absolute Error), MSE (Mean Square Error), and fourth power difference loss (the fourth power of the spectral difference) can be selected sequentially during training. After training with MAE loss, the neural network model achieves high noise reduction, but the degree of speech distortion is relatively large. Therefore, MSE loss is used to fine-tune the distorted part. After fine-tuning, further optimization training is performed with fourth power difference loss, thereby achieving a balance between noise reduction and speech distortion in the trained speech denoising network.

[0064] In another case, if the function value is less than the loss threshold, it indicates that the neural network model has converged. In this case, the neural network model can be used as a speech denoising network.

[0065] Through the above measures, the speech data is first converted into time-frequency domain features during training, so that the neural network model can process the language data as time-frequency domain features more comprehensively. Through multi-stage loss optimization, the loss function is switched during training to optimize features with different numerical ranges, so that the trained speech denoising network can reduce the degree of distortion while achieving a sufficiently good denoising effect.

[0066] See Figure 4 This is a schematic diagram of the training process of a speech denoising network provided in an embodiment of this application, as shown below. Figure 4As shown, the sample speech data can first be converted into sample time-frequency domain features. Then, a neural network model is used to perform forward computation on the noisy time-frequency domain features to obtain the predicted time-frequency domain features. The difference between the predicted and clean time-frequency domain features is calculated using a loss function to obtain the function value, and the model parameters of the neural network model are updated with this function value. It is then determined whether the function value is less than the loss threshold. If not, the forward computation step is returned, and the loss function is switched periodically during training. If so, the neural network model can be considered to have converged, and it can be used as the speech denoising network.

[0067] In one embodiment, a sample dataset can be constructed before training the neural network model. The server can acquire multiple clean speech data sets and, after copying each clean speech data set, add noise data to the copied clean speech data. This noise data can be randomly generated, or it can be pre-configured according to subsequent application scenarios. After adding the noise data, noisy speech data can be obtained.

[0068] The server can use each clean speech data point as a sample label for its corresponding noisy speech data, thus forming a sample speech data set. For example, if the clean speech data is denoted as A, and the generated noisy speech data is denoted as A', then A-A' constitutes a sample speech data set.

[0069] The server can construct a sample dataset based on multiple noisy speech data carrying sample labels, and then train a speech denoising network based on the sample dataset.

[0070] In one embodiment, the network layers in the neural network model can be connected via residual connections. See also Figure 5 This is a schematic diagram of the architecture of a neural network model provided in an embodiment of this application, as shown below. Figure 5 As shown, the neural network model may include a first fully connected layer, a first feature processing module, a second feature processing module, a third feature processing module, and a second fully connected layer. The first, second, and third feature processing modules can be any of CNN (Convolutional Neural Networks) or RNN (Recurrent Neural Networks). Since the neural network model needs to process time-frequency domain features, an RNN is preferred for the feature processing module. For example, the first, second, and third feature processing modules can be any of LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit).

[0071] The first fully connected layer is connected to the first feature processing module. The first fully connected layer, the first feature processing module, and the second feature processing module are residually connected. The first fully connected layer, the first feature processing module, the second feature processing module, and the third feature processing module are residually connected. The third feature processing module is connected to the second fully connected layer.

[0072] In this case, training the residual noise in each layer through multi-layer residual connections helps the speech denoising network achieve a better level of noise reduction.

[0073] In one embodiment, after training a speech denoising model, the model can be applied for denoising. The server can acquire the speech data to be processed, which may belong to the same audio format as the sample speech data. A short-time Fourier transform is performed on the speech data to be processed to obtain the time-frequency domain features. These features are then input into the speech denoising network for forward computation to obtain the denoised time-frequency domain features, which include a frequency domain mask that occludes noise information. The server can then perform an inverse Fourier transform on the denoised time-frequency domain features to obtain the denoised speech data.

[0074] The speech denoising model trained by the scheme in this application can achieve a balance between noise reduction and speech distortion. Therefore, the denoised speech data has both good noise reduction effect and is not too distorted.

[0075] Figure 6 This is a block diagram of a training device for a speech denoising network according to an embodiment of the present invention, as shown below. Figure 6 As shown, the device may include:

[0076] The transformation module 610 is used to perform a short-time Fourier transform on the sample speech data in the sample dataset to obtain the sample time-frequency domain features; wherein, the sample speech data includes noisy speech data and clean speech data corresponding to the noisy speech data, and the sample time-frequency domain features include the noise time-frequency domain features obtained by transforming the noisy speech data and the clean time-frequency domain features obtained by transforming the clean speech data.

[0077] The calculation module 620 is used to calculate the time-frequency domain features of the noise through a neural network model to obtain the predicted time-frequency domain features;

[0078] The adjustment module 630 is used to evaluate the difference between the predicted time-frequency domain features and the clean time-frequency domain features through a loss function, obtain a function value, and adjust the model parameters of the neural network model based on the function value;

[0079] The judgment module 640 is used to determine whether the function value is less than a preset loss threshold; wherein, the loss function is switched periodically during the training process.

[0080] The determination module 650 is used to determine if the neural network model has converged, thereby obtaining the speech denoising network.

[0081] In one embodiment, the device further includes:

[0082] The return module 660 is used to return the step of calculating the time-frequency domain features of the noise through a neural network model to obtain the predicted time-frequency domain features if the function value is not less than the loss threshold.

[0083] In one embodiment, the device further includes:

[0084] The generation module 670 is used to acquire multiple clean speech data and generate corresponding noisy speech data for each clean speech data; wherein, the noisy speech data includes clean speech data and noise data; each clean speech data is used as a sample label for its corresponding noise speech data; and the sample dataset is constructed based on multiple noisy speech data carrying sample labels.

[0085] In one embodiment, the neural network model includes a first fully connected layer, a first feature processing module, a second feature processing module, a third feature processing module, and a second fully connected layer; wherein the first fully connected layer is connected to the first feature processing module, the first fully connected layer, the first feature processing module, and the second feature processing module are residually connected, the first fully connected layer, the first feature processing module, the second feature processing module, and the third feature processing module are residually connected, and the third feature processing module is connected to the second fully connected layer.

[0086] In one embodiment, the first feature processing module, the second feature processing module, and the third feature processing module are recurrent neural networks.

[0087] In one embodiment, the adjustment module 630 is also configured to sequentially select MAE loss, MSE loss and fourth power difference loss during the training process.

[0088] In one embodiment, the device further includes:

[0089] The transformation module 610 is also used to perform a short-time Fourier transform on the speech data to be processed to obtain the time-frequency domain features to be processed.

[0090] The calculation module 620 is also used to calculate the time-frequency domain features to be processed through the speech denoising network to obtain the denoised time-frequency domain features;

[0091] The transformation module 610 is also used to perform inverse Fourier transform on the noise-reduced time-frequency domain features to obtain noise-reduced speech data.

[0092] The specific implementation process of the functions and roles of each module in the above-mentioned device can be found in the implementation process of the corresponding steps in the training method of the above-mentioned speech denoising network, and will not be repeated here.

[0093] The apparatuses and methods disclosed in the several embodiments provided in this application can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0094] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0095] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A training method for a speech denoising network, characterized in that, include: A short-time Fourier transform is performed on the sample speech data in the sample dataset to obtain the sample time-frequency domain features; wherein, the sample speech data includes noisy speech data and clean speech data corresponding to the noisy speech data, and the sample time-frequency domain features include the noise time-frequency domain features obtained by transforming the noisy speech data and the clean time-frequency domain features obtained by transforming the clean speech data. The predicted time-frequency domain features are obtained by calculating the time-frequency domain features of the noise using a neural network model. The difference between the predicted time-frequency domain features and the clean time-frequency domain features is evaluated by a loss function to obtain a function value, and the model parameters of the neural network model are adjusted based on the function value. Determine whether the function value is less than a preset loss threshold; wherein, the loss function is switched periodically during the training process. If so, determine that the neural network model has converged to obtain the speech denoising network; The loss function that is switched in stages during the training process includes: MAE loss, MSE loss and fourth power difference loss are selected in sequence during the training process.

2. The method according to claim 1, characterized in that, The method further includes: If the function value is not less than the loss threshold, return to the step of calculating the noise time-frequency domain features through a neural network model to obtain the predicted time-frequency domain features.

3. The method according to claim 1, characterized in that, Before performing a short-time Fourier transform on the sample speech data in the sample dataset to obtain the sample time-frequency domain features, the method further includes: Multiple clean speech data are acquired, and corresponding noisy speech data is generated for each clean speech data; wherein, the noisy speech data includes clean speech data and noise data; Each clean speech data point is used as a sample label for its corresponding noisy speech data. The sample dataset is constructed based on multiple noisy speech data carrying sample labels.

4. The method according to claim 1, characterized in that, The neural network model includes a first fully connected layer, a first feature processing module, a second feature processing module, a third feature processing module, and a second fully connected layer; wherein, the first fully connected layer is connected to the first feature processing module, the first fully connected layer, the first feature processing module, and the second feature processing module are residually connected, the first fully connected layer, the first feature processing module, the second feature processing module, and the third feature processing module are residually connected, and the third feature processing module is connected to the second fully connected layer.

5. The method according to claim 4, characterized in that, The first feature processing module, the second feature processing module, and the third feature processing module are recurrent neural networks.

6. The method according to claim 1, characterized in that, After obtaining the speech denoising network, the method further includes: The speech data to be processed is subjected to a short-time Fourier transform to obtain the time-frequency domain features to be processed. The speech denoising network is used to calculate the time-frequency domain features to be processed, and the denoised time-frequency domain features are obtained. The denoised time-frequency domain features are subjected to inverse Fourier transform to obtain the denoised speech data.

7. A training device for a speech denoising network, characterized in that, include: The transformation module is used to perform short-time Fourier transform on the sample speech data in the sample dataset to obtain the sample time-frequency domain features; wherein, the sample speech data includes noisy speech data and clean speech data corresponding to the noisy speech data, and the sample time-frequency domain features include the noise time-frequency domain features obtained by transforming the noisy speech data, and the clean time-frequency domain features obtained by transforming the clean speech data. The calculation module is used to calculate the time-frequency domain features of the noise using a neural network model to obtain the predicted time-frequency domain features; The adjustment module is used to evaluate the difference between the predicted time-frequency domain features and the clean time-frequency domain features through a loss function, obtain a function value, and adjust the model parameters of the neural network model based on the function value; The judgment module is used to determine whether the function value is less than a preset loss threshold; wherein, the loss function is switched periodically during the training process. A determination module is used to determine if the neural network model has converged, thereby obtaining a speech denoising network. The judgment module is used to select MAE loss, MSE loss and fourth power difference loss in sequence during the training process.

8. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to execute the training method of the speech denoising network according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that can be executed by a processor to complete the training method of the speech denoising network according to any one of claims 1-6.

Images