A training method for a denoising model, a denoising method, a device and a storage medium

Abstract

The present application discloses a training method, a noise reduction method, a device and a storage medium for a noise reduction model, the method comprising: inputting noisy speech data into the noise reduction model to obtain noise-reduced speech data; the noise reduction model comprises an encoder and a first decoder, and the training method of the noise reduction model comprises: obtaining multiple training samples; processing the sample noisy speech data based on the encoder to obtain first sample feature data; inputting the first sample feature data into the first decoder to obtain estimated speech data; inputting the first sample feature data into the second decoder to obtain estimated text data; based on the loss between the estimated speech data and the first sample clean speech data and the loss between the estimated text data and the first sample clean text data, adjusting the model parameters of the noise reduction model until the preset training end conditions are met to obtain a trained noise reduction model. In the above manner, the present application can improve the effect of speech noise reduction.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a training method for a noise reduction model, a noise reduction method, a device and a storage medium. Background Art

[0002] Speech noise reduction methods are mainly divided into two categories. One is based on traditional statistical signal processing methods. This type of method has a good effect on stationary noise (such as white noise, etc.), and the algorithm is controllable and stable. However, for non-stationary burst noise, the noise reduction effect is poor. Although the controllability and stability are good, the effect is usually lower than expected in real application scenarios; the other speech noise reduction method is a supervised regression training method based on neural networks. With the complexity of the model and the improvement of hardware computing power, the noise reduction effect is becoming more and more difficult to optimize and improve. In addition, the characteristics of some pronunciation phonemes in speech at the signal level are similar to noise, which is difficult to distinguish and solve through network learning, resulting in serious distortion of speech after noise reduction. Summary of the invention

[0003] The present application provides a noise reduction model training method, a noise reduction method, a device and a storage medium, which can improve the effect of speech noise reduction.

[0004] To solve the above technical problems, the technical solution adopted in the present application is: to provide a denoising method, the method comprising: obtaining noisy speech data, inputting the noisy speech data into a denoising model, and obtaining denoised speech data; wherein the denoising model comprises an encoder and a first decoder, and the training method of the denoising model comprises: obtaining multiple training samples, the training samples comprising sample noisy speech data and first sample clean speech data and first sample clean text data corresponding to the sample noisy speech data; processing the sample noisy speech data based on the encoder to obtain first sample feature data; inputting the first sample feature data into the first decoder to obtain estimated speech data; inputting the first sample feature data into the second decoder to obtain estimated text data; based on the loss between the estimated speech data and the first sample clean speech data and the loss between the estimated text data and the first sample clean text data, adjusting the model parameters of the denoising model until the preset training end conditions are met to obtain a trained denoising model.

[0005] To solve the above technical problems, the technical solution adopted in the present application is: to provide a training method for a denoising model, the denoising model includes an encoder and a first decoder, the method including: obtaining multiple training samples, the training samples including sample noisy speech data and first sample clean speech data and first sample clean text data corresponding to the sample noisy speech data; processing the sample noisy speech data based on the encoder to obtain first sample feature data; inputting the first sample feature data into the first decoder to obtain estimated speech data; inputting the first sample feature data into the second decoder to obtain estimated text data; based on the loss between the estimated speech data and the first sample clean speech data and the loss between the estimated text data and the first sample clean text data, adjusting the model parameters of the denoising model until the preset training end conditions are met to obtain the trained denoising model.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide a denoising device, which comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the training method of the denoising model in the above technical solution or the denoising method in the above technical solution.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the training method of the denoising model in the above technical solution or the denoising method in the above technical solution.

[0008] Through the above scheme, the beneficial effect of the present application is: first obtain noisy speech data, then input the noisy speech data into the denoising model to obtain the denoised speech data; the denoising model includes an encoder and a first decoder, and the training process of the denoising model includes: obtain training samples, the training samples include sample noisy speech data, first sample clean text data and first sample clean speech data; then use the encoder to process the sample noisy speech data to generate first sample feature data; use the first decoder to process the first sample feature data to obtain estimated speech data of the sample noisy speech data; and use the second encoder to process the first sample feature data to obtain estimated text data of the sample noisy speech data; in the technical scheme provided by the present application, the training samples include sample noisy speech data, first sample clean speech data and first sample clean text data. Due to the text information, the denoising model can learn to classify the noisy speech data, and can distinguish the noise-like phoneme segments in the noisy speech data from the real noise, reduce the erroneous elimination of consonant segments by the denoising model, and improve the effect of speech denoising. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0010] Figure 1 It is a flowchart of an embodiment of a training method for a denoising model provided in the present application;

[0011] Figure 2 It is a flowchart of another embodiment of the training method of the denoising model provided by the present application;

[0012] Figure 3 yes Figure 2 A schematic diagram of the flow chart of step 21 in the embodiment shown;

[0013] Figure 4 It is a structural schematic diagram of the noise reduction model provided in this application;

[0014] Figure 5 yes Figure 2 A schematic diagram of the flow chart of step 23 in the embodiment shown;

[0015] Figure 6 It is a flowchart of an embodiment of a noise reduction method provided by the present application;

[0016] Figure 7 It is a structural schematic diagram of an embodiment of a noise reduction device provided by the present application;

[0017] Figure 8 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0018] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0019] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0020] It should be noted that the terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0021] The speech denoising method based on neural network needs to establish a set of network model structures. Through supervised training, the model gradually learns certain characteristics of clean speech and noise in noisy speech, so as to extract the characteristics of clean speech and use them as the output of the network model. There are many common model structures, such as: Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short Term Memory Networks (Long Short Term Memory networks), Gated Recurrent Unit (GRU), Convolutional Recurrent Network (CRN) or Temporal Convolutional Network (TCN) and other network structures.

[0022] However, the above network structures all learn the mapping relationship between noisy speech and clean speech through supervised training of noisy speech and clean speech, and the mapping relationship is one-to-one, that is, each input data point has a corresponding output, for example: input a 1-second noisy speech signal, and the output is an estimated clean signal of equal length. Moreover, the goal of network training is to make the error between the estimated clean signal and the clean signal in the training data smaller and smaller; therefore, the cost function used in training the network is L1 loss (Loss) or mean square error (MSE), etc., which are essentially to make the estimated signal and the target signal closer and closer. Therefore, the speech denoising task is usually treated as a regression problem in the NN field; however, there are cases where clear consonants such as "s", "t" or "f" are eliminated in the audio output of the NN-based denoising model, resulting in severe speech distortion.

[0023] Based on the above-mentioned problems of NN, this application changes the training method and introduces the classification of the text content of the speech itself, so that the model can learn the characteristics of the phonemes themselves from the language level, thereby assisting in distinguishing noise characteristics and speech characteristics, especially the differences between some speech phoneme features similar to noise, thereby improving the noise reduction effect of NN. The technical solution adopted in this application is described in detail below.

[0024] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of a method for training a denoising model provided in the present application, the method comprising:

[0025] Step 11: Obtain multiple training samples.

[0026] First, a training sample is obtained, which includes sample noisy speech data and first sample clean speech data and first sample clean text data corresponding to the sample noisy speech data, and the first sample clean speech data corresponds to the first sample clean text data; the first sample noise data and the first sample clean speech data are synthesized into noisy speech data, so that the sample noisy speech data can be used to train the noise reduction model, that is, the sample noisy speech data is formed by the superposition of the first sample clean speech data and the first sample noise data.

[0027] Furthermore, the first sample noise data should cover as many scenarios as possible to improve the noise generalization ability of the noise reduction model; the first sample clean speech data is a clean speech signal that does not contain noise, for example: the first sample clean speech data is speech with a signal-to-noise ratio higher than a threshold (for example: 30dB or 80dB); the language type of the first sample clean text data can be Chinese, English or other language types.

[0028] Step 12: Process the sample noisy speech data based on the encoder to obtain first sample feature data.

[0029] The denoising model is used to process the training samples to generate first sample feature data; extract and classify the first sample feature data to obtain estimated speech data of the sample noisy speech data and estimated text data of the sample noisy speech data. Specifically, the sample noisy speech data in the training samples can be directly input into the encoder so that the encoder encodes the sample noisy speech data to obtain the first sample feature data; or feature extraction is performed on the sample noisy clean speech data to obtain third sample feature data, and the third sample feature data is input into the encoder.

[0030] It can be understood that during the training process, the denoising model includes the encoder, the first decoder and the second decoder; but after the training, the denoising model only includes the encoder and the first decoder.

[0031] Step 13: Input the first sample feature data into the first decoder to obtain estimated speech data.

[0032] The first sample feature data is subjected to feature extraction processing to extract speech features (ie, estimated speech data) from the sample noisy speech data.

[0033] Step 14: Input the first sample feature data into the second decoder to obtain estimated text data.

[0034] The first sample feature data is classified to obtain text features (ie, estimated text data) in the corresponding sample noisy speech data.

[0035] Step 15: Based on the loss between the estimated speech data and the first sample clean speech data and the loss between the estimated text data and the first sample clean text data, the model parameters of the denoising model are adjusted until the preset training end conditions are met to obtain a trained denoising model.

[0036] After obtaining the estimated speech data and the estimated text data, the loss between the estimated speech data and the first sample clean speech data and the loss between the estimated text data and the first sample clean text data can be calculated first, and then based on the two losses, it is determined whether the training end condition (i.e., the preset training end condition) is met. If the preset training end condition is not met, the model parameters of the denoising model are adjusted until the preset training end condition is met to obtain a trained denoising model. By continuously using training samples to train the denoising model and modifying the model parameters of the denoising model, a trained denoising model is finally obtained.

[0037] In the training scheme of the noise reduction model provided in the present embodiment, the training samples include not only the sample noisy speech data, but also the first sample clean text data corresponding to the clean first sample clean speech data in the sample noisy speech data, that is, the first sample clean text data corresponding to the first sample clean speech data is also added to the training samples as an important training target; because of the language / text information, the noise reduction model can learn to classify phonemes, and can distinguish the noise-like phoneme segments in the sample noisy speech data from the real noise features according to the context relationship, thereby reducing the erroneous elimination of consonant segments in the noise-reduced speech data by the noise reduction model, and improving the effect of speech noise reduction.

[0038] See also Figure 2 , Figure 2 : is a flow chart of another embodiment of a method for training a denoising model provided in the present application, the method comprising:

[0039] Step 21: Obtain first sample noise data and first sample clean speech data.

[0040] In order to obtain the data required for training the noise reduction model, the second sample noise data and the second sample clean speech data can be first obtained, where the second sample noise data is the obtained noise data, and the second sample clean speech data is the obtained speech data without noise; then the second sample noise data and the second sample clean speech data are preprocessed respectively to obtain the first sample noise data and the first sample clean speech data.

[0041] In a specific embodiment, it is possible to use Figure 3 The scheme shown is preprocessed:

[0042] Step 31: Transcribe the second sample clean speech data to generate second sample clean character data.

[0043] It can be determined whether the format of the second sample noise data and the second sample clean voice data is a preset format. When the format of the second sample clean voice data is not a preset format, the second sample clean voice data is format-converted so that the format of the converted second sample clean voice data (i.e., the first sample clean voice data) is a preset format; when the format of the second sample noise data is not a preset format, the second sample noise data is format-converted so that the format of the converted second sample noise data (i.e., the first sample noise data) is a preset format. For example, the second sample noise data and the clean second sample clean voice data are format-unified and converted into single-channel pulse code modulation (PCM) data with a sampling rate of 16KHz and a quantization accuracy of float32 (the range of the quantized signal is between ±1). Then, the audio transcription technology in the related art is used to process the second sample clean voice data to obtain the corresponding second sample clean character data.

[0044] Step 32: Delete punctuation marks in the second sample clean character data, and convert the language type of each character in the second sample clean character data into a preset language type to obtain the first sample clean character data.

[0045] Each clean second sample clean voice data has a corresponding text. For ease of processing, the language type of each character in the second sample clean character data can be converted into a preset language type, which is a language type set according to the application scenario or experience; for example: taking the preset language type as Chinese as an example, all numbers in the second sample clean character data can be regularized into Chinese characters with the same pronunciation, and all punctuation marks can be deleted.

[0046] Step 33: De-duplicate all the first sample clean character data to obtain third sample clean character data.

[0047] All characters contained in the first sample clean character data are put together for deduplication to obtain third sample clean character data; or a set of all characters represented by a preset language type can be directly used as the third sample clean character data, for example, a set of all Chinese characters (for example: all Chinese characters in a dictionary) can be directly used as the third sample clean character data.

[0048] Step 34: assign a dictionary ID to each character in the third sample clean character data to construct a text dictionary; assign a dictionary ID to the first sample clean character data based on the text dictionary to obtain the first sample clean text data.

[0049] After obtaining the third sample clean character data, a correspondence between the characters in the third sample clean character data and a dictionary identification (ID) can be established, that is, a dictionary ID is created for each character in the third sample clean character data to form a text dictionary, the text dictionary includes each character in the third sample clean character data and a dictionary ID corresponding to the character, the dictionary ID can be a label for each character to distinguish different characters, that is, the character and the dictionary ID correspond one to one, for example: assuming that there are a total of 4 characters in the third sample clean character data, the dictionary IDs of these 4 characters can be 0, 1, 2 and 3 respectively. After generating the text dictionary, a dictionary ID is assigned to all characters in the first sample clean text data according to the text dictionary.

[0050] Furthermore, all characters in the third sample clean character data, preset sentence start characters and preset sentence end characters can be sorted to generate a text character sequence; a dictionary ID is assigned to each character in the text character sequence, for example: the dictionary ID corresponding to each character in the text character sequence is set to a number, and the dictionary ID corresponds to the character one by one.

[0051] In a specific embodiment, the preset sentence start character and the preset sentence end character are respectively " <s> "and"< / s> ", plus the dictionary ID corresponding to the first sample clean text data <s> "and"< / s> " These two symbols are used as the beginning and end of the sentence, and then these two symbols and all Chinese characters are sorted (for example, sorted by the first letter of the pinyin), and dictionary IDs are assigned starting from 0 to serve as the classification dictionary for subsequent training.

[0052] Step 22: Superimpose the first sample noise data and the first sample clean speech data to obtain sample noisy speech data.

[0053] A first sample clean speech data is selected from all the first sample clean speech data as the speech data to be synthesized; a first sample noise data is selected from all the first sample noise data as the noise data to be synthesized; and then based on a preset signal-to-noise ratio and a preset signal energy, the speech data to be synthesized and the speech data to be synthesized are superimposed to obtain sample noisy speech data.

[0054] Furthermore, it can also be determined whether the length of the noise data to be synthesized is greater than the speech data to be synthesized; if the length of the noise data to be synthesized is greater than the speech data to be synthesized, the noise data to be synthesized is truncated so that the length of the speech data to be synthesized is equal to the length of the truncated noise data to be synthesized. If the length of the noise data to be synthesized is less than the speech data to be synthesized, the noise data to be synthesized is copied, and the noise data to be synthesized and the copied noise data to be synthesized are spliced; then it is determined whether the length of the spliced ​​noise data to be synthesized is greater than the length of the speech data to be synthesized; if the length of the spliced ​​noise data to be synthesized is greater than the length of the speech data to be synthesized, the spliced ​​noise data to be synthesized is truncated so that the length of the speech data to be synthesized is equal to the length of the truncated noise data to be synthesized, that is, when the length of the noise data to be synthesized is less than the speech data to be synthesized, the noise data to be synthesized is repeated several times so that its length exceeds the length of the speech data to be synthesized, and then randomly truncated to be superimposed with the speech data to be synthesized.

[0055] In a specific embodiment, assuming that there are N clean first sample clean voice data in total, all the first sample clean voice data can be looped through K times, and one first sample clean voice data is taken in order each time, recorded as x(t), and a total of K*N pieces are extracted; then one piece is randomly obtained from all the first sample noise data, and then a section of first sample noise data with the same length as the current first sample clean voice data is randomly intercepted therefrom, recorded as n(t); then according to uniform random distribution, a preset signal-to-noise ratio α and a preset signal energy β are specified, and then the signal is superimposed according to the following formula to obtain the sample noisy voice data y(t):

[0056]

[0057]

[0058]

[0059] y'(t)=x(t)+10 c / 10 *n(t) (4)

[0060] σ y' =max(|y'(t)| 2 ) (5)

[0061]

[0062] Each generated sample noisy speech data has corresponding first sample clean speech data and first sample clean text data; each character in the first sample clean text data is mapped to a corresponding dictionary ID according to the text dictionary to form Label_ID, and saved as an array; finally, K*N groups of training samples including y(t), x(t) and Label_ID are formed, and they can be randomly shuffled and saved.

[0063] Step 23: Process the sample noisy speech data based on the encoder to obtain first sample feature data.

[0064] Step 24: Input the first sample feature data into the first decoder to obtain estimated speech data.

[0065] Step 25: Input the first sample feature data into the second decoder to obtain estimated text data.

[0066] Steps 23-25 ​​are the same as steps 12-14 in the above embodiment and will not be described again.

[0067] In a specific embodiment, the estimated speech data is the amplitude spectrum of sample noisy speech data, such as Figure 4 As shown, this embodiment adopts an encoder-decoder (ED) structure as a noise reduction model, which is composed of an encoder and two decoders; specifically, the noise reduction model includes an encoder, a first decoder and a second decoder; the first decoder is used to perform feature extraction processing on the first sample feature data to obtain an amplitude spectrum; then the second decoder is used to perform text classification processing on the first sample feature data to obtain estimated text data. Further, the encoder is used to receive the features of the sample noisy speech data (such as amplitude spectrum features) and output high-dimensional feature information; the first decoder is used to perform signal extraction on the high-dimensional feature information and output an estimated clean speech amplitude spectrum; the second decoder is used to perform text classification on the high-dimensional feature information and output estimated text data of the clean speech.

[0068] Step 26: Calculate the current loss value based on the difference between the dictionary ID in the first sample clean text data and the dictionary ID in the estimated text data and the difference between the first sample clean speech data and the estimated speech data.

[0069] The first sample clean text data and the estimated text data both include a dictionary ID, the first sample clean text data includes multiple frames of clean text signals, and the estimated text data includes multiple frames of estimated text signals; the first sample clean speech data includes multiple frames of clean speech signals, and the estimated speech data includes multiple frames of estimated speech signals.

[0070] In a specific embodiment, it is possible to use Figure 5The solution shown is processed, which specifically includes the following steps:

[0071] Step 51: Calculate the difference between the dictionary ID of each frame of the clean text signal and the dictionary ID of the estimated text signal to obtain a first loss value.

[0072] The first loss value can be calculated by the cross entropy (CE) function. The time corresponding to the current frame is recorded as t, the dictionary ID corresponding to the clean text signal at time t is recorded as x_id(t), and the dictionary ID corresponding to the estimated text signal at time t is recorded as x_id'(t). The CE Loss between x_id(t) and x_id'(t) is calculated to obtain the first loss value.

[0073] Step 52: Calculate the difference between the clean speech signal and the estimated speech signal in each frame to obtain a second loss value.

[0074] The second loss value is calculated by the MSE function. The clean speech signal at time t is recorded as x(t), and the estimated speech signal at time t is recorded as x'(t). The MSE Loss between x(t) and x'(t) is calculated to obtain the second loss value.

[0075] Step 53: Perform a weighted sum of the first loss value and the second loss value to obtain a loss value, and average all loss values ​​to obtain a current loss value.

[0076] The following formula is used to calculate the current loss value:

[0077] L=(1-λ)*L MSE +λ*L CE (7)

[0078] Among them, L is the current loss value, and λ represents the weighted coefficient between MSE Loss and CE Loss, which can be set according to the specific application scenario, for example: 0.2.

[0079] Continue reading Figure 4 The denoising model also includes a self-attention layer, which is connected to the encoder and the second decoder and is used to output second sample feature data; specifically, the first sample feature data includes multiple frames of first sample sub-features, and the second sample feature data can be obtained first, the second sample feature data includes multiple frames of second sample sub-features, and the second sample feature data includes multiple frames of second sample sub-features; then the attention layer is used to process the first sample sub-features of the current frame and the second sample sub-features of the previous frame to generate the current frame state features, and the current frame state features are input into the second decoder.

[0080] Furthermore, the moment corresponding to the previous frame is recorded as the (t-1) moment. The attention layer calculates the attention coefficient between the second sample sub-feature h_att(t-1) output at the previous moment (i.e., the t-1 moment) and the first sample sub-feature h_enc(t) at the t moment, and outputs the current frame state feature c(t). The specific calculation process is as follows:

[0081] (1) Multiply the first weight matrix by the first sample sub-feature at time t to obtain the first sample feature matrix:

[0082] h K (t) = W K *h_enc(t) (8)

[0083] Among them, W K is the first weight matrix, h k (t) is the first sample feature matrix at time t.

[0084] (2) Multiply the second weight matrix by the first sample sub-feature at time t to obtain the second sample feature matrix:

[0085] h V (t) = W V *h_enc(t) (9)

[0086] Among them, W V is the second weight matrix, h V (t) is the second sample feature matrix at time t.

[0087] (3) Multiply the third weight matrix by the second sample sub-feature h_att(t-1) at time (t-1) to obtain the third feature matrix:

[0088] h Q (t) = W Q *h_att(t-1) (10)

[0089] Among them, W Q is the third weight matrix, h Q (t) is the third characteristic matrix at time t.

[0090] (4) The first sample feature matrix, the second sample feature matrix, the third feature matrix and the state features at the previous moment are processed using a logistic regression model to generate the state features at time t:

[0091]

[0092] Among them, softmax() is the logistic regression function, H K T(t) is composed of the first sample feature matrices of multiple moments including moment t (moments before moment t), H V T (t) is composed of the second sample feature matrices of multiple moments including moment t (moments before moment t).

[0093] Understandably, Figure 4 The other parameters in represent the eigenvectors inside the denoising model. Figure 4 The solid line in the middle represents the forward process of model training, and the dotted line represents the gradient backpropagation process of the model. Figure 4 The denoising model shown only provides a basic model framework and training method. The interior of each part can be flexibly adjusted. For example, the encoder uses three stacked bidirectional LSTM layers; the second decoder uses one stacked unidirectional LSTM layer and one stacked CNN layer, and finally connects a linear output layer; the first decoder uses three stacked CNN layers, and finally connects a Liner output layer.

[0094] Step 27: Based on the current loss value, determine whether the preset training end condition is met.

[0095] After calculating the current loss value, determine whether the preset training end conditions are met. The preset training end conditions include: the loss value cannot continue to decrease or the loss value converges, for example: the difference between the previous loss value and the current loss value is less than the set value; or, it can also be determined whether the current loss value is less than the preset loss value, which is a preset loss threshold. If the current loss value is less than the preset loss value, it is determined that the preset training end conditions are met; or, the number of training times reaches the set value (for example: 10,000 training times).

[0096] When the preset training end condition is not met at present, the current loss value is used for back propagation, the model parameters of the denoising model are adjusted by the gradient descent method, training is continued, new training samples are obtained, the above steps of processing the new training samples are executed, and the process returns to step 51 until the preset training end condition is met; it can be understood that the method used for back propagation of the loss value to update the model parameters is similar to the method used for updating the model parameters in the related art, and will not be repeated here.

[0097] This embodiment provides a training method for a NN denoising model based on phoneme feature classification, which is mainly divided into three parts: first, obtaining pure noise data (excluding voice information) and clean voice data with text annotations; second, for the noise data and clean voice data, data enhancement is performed through data simulation to generate training samples that are several times the original data; finally, a denoising network and Loss function are designed for model training. In this embodiment, the text of clean speech is also added to the training samples, and the training samples are constructed using the clean speech signal itself and the text of clean language for training two tasks: the statistical error (MSE Loss) of the speech signal itself and the classification of language information (CE Loss) of the speech content, providing two related constraints belonging to different feature levels for model training. Since MSE Loss and CE Loss are estimated at the same time, the two share an encoder, which allows the model, especially the encoder, to better learn the characteristics of phonemes and noise, provide better feature space distinguishability, improve the noise reduction effect, and reduce the probability of speech distortion in the final output; moreover, the classification and regression tasks share the same encoder structure, allowing the encoder to simulate the auditory characteristics of the human brain, that is, to be able to hear clearly and understand, rather than simply looking for a feature space that minimizes the statistical error at the signal level, and the denoising effect is better.

[0098] See also Figure 6 , Figure 6 : is a flow chart of an embodiment of a noise reduction method provided by the present application, the method comprising:

[0099] Step 61: Acquire noisy speech data.

[0100] The noisy speech data is speech data containing noise that needs to be processed by noise reduction. The noisy speech data can be collected by an audio collection device (such as a microphone) or obtained from an audio database.

[0101] Step 62: Input the noisy speech data into the noise reduction model to obtain the noise-reduced speech data.

[0102] Based on the encoder, the noisy speech data is encoded to obtain current feature data; the current feature data is input into the first decoder to obtain the denoised speech data; specifically, after obtaining the noisy speech data, the feature extraction method in the related technology can be used to perform feature extraction on the noisy speech data to obtain the amplitude spectrum feature of the noisy speech data; then the amplitude spectrum feature is input into the denoising model, so that the denoising model processes the amplitude spectrum feature to generate corresponding denoised speech data, and the denoising model is obtained by the denoising model training method in the above embodiment.

[0103] The denoising method adopted in this embodiment improves the denoising model. In addition to supervised regression training, it assists in classifying the text content of the sample noisy speech data. By simultaneously training two related tasks, the NN denoising effect is improved, which can reduce the probability of certain consonants being eliminated in the speech data that needs denoising.

[0104] See also Figure 7 , Figure 7 It is a structural schematic diagram of an embodiment of a noise reduction device provided in the present application. The noise reduction device 70 includes a memory 71 and a processor 72 connected to each other. The memory 71 is used to store a computer program. When the computer program is executed by the processor 72, it is used to implement the training method of the noise reduction model in the above embodiment or the noise reduction method in the above embodiment.

[0105] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 80 is used to store a computer program 81. When the computer program 81 is executed by the processor, it is used to implement the training method of the denoising model in the above embodiment or the denoising method in the above embodiment.

[0106] The computer-readable storage medium 80 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other media that can store program codes.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A noise reduction method, comprising: Acquire noisy speech data, and input the noisy speech data into a noise reduction model to obtain noise-reduced speech data; The denoising model includes an encoder and a first decoder, and the training method of the denoising model includes: Acquire a plurality of training samples, wherein the training samples include sample noisy speech data and first sample clean speech data and first sample clean text data corresponding to the sample noisy speech data; Processing the sample noisy speech data based on the encoder to obtain first sample feature data; Inputting the first sample feature data into the first decoder to obtain estimated speech data; Inputting the first sample feature data into a second decoder to obtain estimated text data; Based on the loss between the estimated speech data and the first sample clean speech data and the loss between the estimated text data and the first sample clean text data, adjusting the model parameters of the denoising model until a preset training end condition is met, thereby obtaining a trained denoising model; Wherein, both the first sample clean text data and the estimated text data include a dictionary ID, and the method further includes: Calculating a current loss value based on the difference between the dictionary ID in the first sample clean text data and the dictionary ID in the estimated text data and the difference between the first sample clean speech data and the estimated speech data; Based on the current loss value, determining whether the preset training end condition is met; If not, adjust the model parameters and continue training; Wherein, the method further comprises: Acquire second sample noise data and second sample clean speech data; Preprocessing the second sample noise data and the second sample clean speech data respectively to obtain first sample noise data and the first sample clean speech data; performing transcription processing on the second sample clean speech data to generate second sample clean character data; Deleting punctuation marks in the second sample clean character data, and converting the language type of each character in the second sample clean character data into a preset language type, to obtain first sample clean character data; Deduplication processing is performed on all the first sample clean character data to obtain third sample clean character data; Assigning a dictionary ID to each character in the third sample clean character data to construct a text dictionary, wherein the text dictionary includes each character in the third sample clean character data and a dictionary ID corresponding to the character; Based on the text dictionary, a dictionary ID is assigned to the first sample clean character data to obtain the first sample clean text data.

2. The noise reduction method according to claim 1, characterized in that: The first sample clean text data includes multiple frames of clean text signals, the estimated text data includes multiple frames of estimated text signals, the first sample clean voice data includes multiple frames of clean voice signals, the estimated voice data includes multiple frames of estimated voice signals, and the method further includes: Calculating the difference between the dictionary ID of the clean text signal and the dictionary ID of the estimated text signal in each frame to obtain a first loss value; Calculate the difference between the clean speech signal and the estimated speech signal in each frame to obtain a second loss value; Performing a weighted summation on the first loss value and the second loss value to obtain a loss value; All the loss values ​​are averaged to obtain the current loss value.

3. The noise reduction method according to claim 1, characterized in that: The first sample feature data includes multiple frames of first sample sub-features. Before the step of inputting the first sample feature data into the first decoder to obtain estimated speech data, the method includes: Acquire second sample feature data, where the second sample feature data includes multiple frames of second sample sub-features; The first sample sub-feature of the current frame and the second sample sub-feature of the previous frame are processed by the self-attention layer to generate a current frame state feature, and the current frame state feature is input into the second decoder.

4. The noise reduction method according to claim 1, characterized in that: The step of obtaining a plurality of training samples comprises: Acquire first sample noise data and first sample clean speech data; The first sample noise data and the first sample clean speech data are superimposed to obtain the noisy speech data.

5. The noise reduction method according to claim 4, characterized in that: The step of assigning a dictionary ID to each character in the third sample clean character data comprises: Sort all characters, preset sentence start characters, and preset sentence end characters in the third sample clean character data to generate a text character sequence; The dictionary ID corresponding to each character in the text character sequence is set to a number, and the dictionary ID corresponds to the character one by one.

6. The noise reduction method according to claim 5, characterized in that: Before the step of transcribing the second sample clean voice data to generate the second sample clean character data, the method includes: When the format of the second sample clean voice data is not a preset format, performing format conversion processing on the second sample clean voice data so that the format of the first sample clean voice data is the preset format; When the format of the second sample noise data is not the preset format, format conversion processing is performed on the second sample noise data so that the format of the first sample noise data is the preset format.

7. The noise reduction method according to claim 4, characterized in that: The step of performing superposition processing on the first sample noise data and the first sample clean speech data to obtain the noisy speech data comprises: Selecting a first sample clean speech data from all the first sample clean speech data as the speech data to be synthesized; Selecting one piece of first sample noise data from all the first sample noise data as the noise data to be synthesized; Based on a preset signal-to-noise ratio and a preset signal energy, the speech data to be synthesized and the speech data to be synthesized are superimposed to obtain the sample noisy speech data.

8. The noise reduction method according to claim 7, characterized in that: Before the step of superimposing the speech data to be synthesized with the speech data to be synthesized based on the preset signal-to-noise ratio and the preset signal energy, the method includes: Determining whether the length of the noise data to be synthesized is greater than the speech data to be synthesized; If the length of the noise data to be synthesized is greater than the speech data to be synthesized, the noise data to be synthesized is truncated so that the length of the speech data to be synthesized is equal to the length of the truncated noise data to be synthesized; If the length of the noise data to be synthesized is smaller than the speech data to be synthesized, the noise data to be synthesized is copied and the noise data to be synthesized is spliced ​​with the copied noise data to be synthesized; whether the length of the spliced ​​noise data to be synthesized is larger than the length of the speech data to be synthesized is determined; if the length of the spliced ​​noise data to be synthesized is larger than the length of the speech data to be synthesized, the spliced ​​noise data to be synthesized is truncated so that the length of the speech data to be synthesized is equal to the length of the truncated noise data to be synthesized.

9. The noise reduction method according to claim 1, characterized in that: The step of processing the sample noisy speech data based on the encoder comprises: Inputting the sample noisy speech data into the encoder; or Perform feature extraction processing on the sample noisy speech data to obtain third sample feature data, and input the third sample feature data into the encoder.

10. The noise reduction method according to claim 1, characterized in that: The step of inputting the noisy speech data into a noise reduction model to obtain the noise-reduced speech data comprises: Performing encoding processing on the noisy speech data based on the encoder to obtain current feature data; The current feature data is input into the first decoder to obtain the noise-reduced speech data.

11. A method for training a denoising model, characterized in that: The denoising model includes an encoder and a first decoder, and the method includes: Acquire a plurality of training samples, wherein the training samples include sample noisy speech data and first sample clean speech data and first sample clean text data corresponding to the sample noisy speech data; Processing the sample noisy speech data based on the encoder to obtain first sample feature data; Inputting the first sample feature data into the first decoder to obtain estimated speech data; Inputting the first sample feature data into a second decoder to obtain estimated text data; Based on the loss between the estimated speech data and the first sample clean speech data and the loss between the estimated text data and the first sample clean text data, adjusting the model parameters of the denoising model until a preset training end condition is met, thereby obtaining a trained denoising model; Wherein, both the first sample clean text data and the estimated text data include a dictionary ID, and the method further includes: Calculating a current loss value based on the difference between the dictionary ID in the first sample clean text data and the dictionary ID in the estimated text data and the difference between the first sample clean speech data and the estimated speech data; Based on the current loss value, determining whether the preset training end condition is met; If not, adjust the model parameters and continue training; Wherein, the method further comprises: Acquire second sample noise data and second sample clean speech data; Preprocessing the second sample noise data and the second sample clean speech data respectively to obtain first sample noise data and the first sample clean speech data; performing transcription processing on the second sample clean speech data to generate second sample clean character data; Deleting punctuation marks in the second sample clean character data, and converting the language type of each character in the second sample clean character data into a preset language type, to obtain first sample clean character data; Deduplication processing is performed on all the first sample clean character data to obtain third sample clean character data; Assigning a dictionary ID to each character in the third sample clean character data to construct a text dictionary, wherein the text dictionary includes each character in the third sample clean character data and a dictionary ID corresponding to the character; Based on the text dictionary, a dictionary ID is assigned to the first sample clean character data to obtain the first sample clean text data.

12. A noise reduction device, characterized in that: It comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the denoising method described in any one of claims 1 to 10 or the denoising model training method described in claim 11.

13. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the denoising method described in any one of claims 1 to 10 or the denoising model training method described in claim 11.

Images