A voice noise reduction method, device, storage medium and equipment

Abstract

This application discloses a speech denoising method, apparatus, storage medium, and device. The method includes: firstly, acquiring the target speech to be denoised, and then generating the amplitude spectrum features of the target speech; next, inputting the amplitude spectrum features of the target speech into a pre-constructed speech denoising model to identify noisy speech and clean speech in the target speech. The speech denoising model is trained using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function. It is evident that this application first constructs a speech denoising model using noisy speech data of different types and / or scenarios, along with the first, second, and third objective functions, and then inputs the amplitude spectrum features of the target speech into the denoising model for denoising processing. This allows for more targeted estimation of noise in the signal, which is then removed from the target speech to obtain clean speech, effectively improving the denoising effect.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech noise reduction method, apparatus, storage medium and device. Background Technology

[0002] With the continuous breakthroughs in artificial intelligence technology and the increasing popularity of various smart terminal devices, human-computer interaction is occurring more and more frequently in people's daily work and life. As the next generation of human-computer interaction, voice interaction can bring great convenience to people's lives. In particular, with increasingly complex noisy environments, the need for noise reduction of voice signals is becoming more and more urgent.

[0003] Traditional speech denoising methods typically fall into two categories: The first is based on traditional statistical signal processing, which focuses on estimating the spectral characteristics of the noise. This method performs well for stationary noise, such as white noise, and is algorithmically controllable and stable. However, its effectiveness is significantly lower for non-stationary burst noise. The second type is based on supervised regression training using neural networks. This method treats the denoising task as a regression problem. Regardless of the model structure, the ultimate goal remains minimizing the second-order variance between the output estimated signal and the target signal, still involving signal-level information. Therefore, as the models become increasingly complex, the denoising performance of this type of method becomes increasingly difficult to optimize and improve. Summary of the Invention

[0004] The main objective of this application is to provide a speech noise reduction method, apparatus, storage medium, and device that can effectively improve the noise reduction effect during speech noise reduction.

[0005] This application provides a speech noise reduction method, including:

[0006] Obtain the target speech to be denoised;

[0007] Generate the amplitude spectrum features of the target speech;

[0008] The amplitude spectrum features of the target speech are input into a pre-built speech denoising model to identify noisy speech and clean speech in the target speech;

[0009] The speech denoising model includes an encoding layer, a convolutional layer, an attention layer, and a decoding layer; the speech denoising model is trained using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function.

[0010] In one possible implementation, the speech denoising model is constructed as follows:

[0011] Acquire noisy speech samples of different types and / or scenarios; and acquire clean speech samples;

[0012] The noisy speech sample and the clean speech sample are superimposed to obtain the sample speech, and the sample speech is numbered according to the classification of the noisy speech sample.

[0013] Generate the amplitude spectrum features of the sample speech;

[0014] The initial speech denoising model is trained using the amplitude spectrum features of the sample speech, the first objective function, the second objective function, the third objective function, and the clean sample speech to generate a speech denoising model.

[0015] In one possible implementation, the first objective function is a mean squared loss function; the mean squared loss function reduces the error between the clean speech output by the speech denoising model and the clean sample speech; the second objective function is a cosine similarity loss function; the cosine similarity loss function is applied to the decoding layer to reduce the similarity between noise sample speech of different types and / or scenarios, the similarity between noise sample speech of the same type and / or scenario, and to optimize each sample speech towards a direction where channel information cannot be distinguished; the third objective function is a mean absolute error loss function; the mean absolute error loss function is applied to the attention layer to allocate the weighting coefficients generated by the attention layer to the correct output features of the convolutional layer.

[0016] In one possible implementation, before performing the superposition process on the noisy sample speech and the clean sample speech to obtain the sample speech, the method further includes:

[0017] The noise sample speech and the clean sample speech are subjected to format unification processing to obtain preprocessed noise sample speech and clean sample speech;

[0018] The process of superimposing the noisy sample speech and the clean sample speech to obtain sample speech includes:

[0019] The preprocessed noisy speech sample and the clean speech sample are superimposed to obtain the sample speech.

[0020] In one possible implementation, the method further includes:

[0021] Acquire noisy verification speech of different types and / or scenarios; and acquire clean verification speech;

[0022] The noisy verification speech and the clean verification speech are superimposed to obtain the verification speech, and the verification speech is numbered according to the classification of the noisy verification speech.

[0023] Generate the amplitude spectrum features of the verification speech;

[0024] The amplitude spectrum features of the verification speech are input into the speech denoising model to obtain the noisy speech prediction results and clean speech prediction results in the verification speech;

[0025] When the clean speech prediction result of the verification speech is inconsistent with the clean verification speech, the verification speech is used again as the sample speech to update the speech denoising model.

[0026] In one possible implementation, the signal-to-noise ratio of the clean sample speech is not less than 25dB.

[0027] In one possible implementation, the step of inputting the amplitude spectrum features of the target speech into a pre-built speech denoising model to identify noisy and clean speech in the target speech includes:

[0028] The amplitude spectrum features of the target speech are input into the coding layer of the speech denoising model to obtain a high-dimensional speech representation vector of the target speech;

[0029] The high-dimensional speech representation vector of the target speech is input into the convolutional layer of the speech denoising model to obtain the feature vectors of the target speech under different types and / or scenarios.

[0030] The high-dimensional speech representation vector of the target speech and the feature vectors of the target speech under different types and / or scenarios are input into the attention layer of the speech denoising model for weighted processing to obtain the weighted speech representation vector.

[0031] The weighted speech representation vector is input into the decoding layer of the speech denoising model for decoding processing to obtain the amplitude spectrum features of the noisy speech in the target speech. Then, using the amplitude spectrum features of the target speech and the amplitude spectrum features of the noisy speech, the amplitude spectrum features of the clean speech in the target speech are determined, thereby identifying the noisy speech and the clean speech in the target speech.

[0032] This application also provides a speech noise reduction device, including:

[0033] The first acquisition unit is used to acquire the target speech to be denoised.

[0034] The first generation unit is used to generate the amplitude spectrum features of the target speech;

[0035] The noise reduction unit is used to input the amplitude spectrum features of the target speech into a pre-built speech noise reduction model to identify the noisy speech and clean speech in the target speech;

[0036] The speech denoising model includes an encoding layer, a convolutional layer, an attention layer, and a decoding layer; the speech denoising model is trained using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function.

[0037] In one possible implementation, the device further includes:

[0038] The second acquisition unit is used to acquire noisy speech samples of different types and / or scenarios; and to acquire clean speech samples.

[0039] The first superposition unit is used to superimpose the noise sample speech and the clean sample speech to obtain sample speech, and to number the sample speech according to the classification of the noise sample speech;

[0040] The second generation unit is used to generate the amplitude spectrum features of the sample speech;

[0041] The training unit is used to train the initial speech denoising model using the amplitude spectrum features of the sample speech, the first objective function, the second objective function, the third objective function, and the clean sample speech, to generate a speech denoising model.

[0042] In one possible implementation, the first objective function is a mean squared loss function; the mean squared loss function reduces the error between the clean speech output by the speech denoising model and the clean sample speech; the second objective function is a cosine similarity loss function; the cosine similarity loss function is applied to the decoding layer to reduce the similarity between noise sample speech of different types and / or scenarios, the similarity between noise sample speech of the same type and / or scenario, and to optimize each sample speech towards a direction where channel information cannot be distinguished; the third objective function is a mean absolute error loss function; the mean absolute error loss function is applied to the attention layer to allocate the weighting coefficients generated by the attention layer to the correct output features of the convolutional layer.

[0043] In one possible implementation, the device further includes:

[0044] The preprocessing unit is used to perform format unification processing on the noise sample speech and the clean sample speech to obtain preprocessed noise sample speech and clean sample speech.

[0045] The first superposition unit is specifically used for:

[0046] The preprocessed noisy speech sample and the clean speech sample are superimposed to obtain the sample speech.

[0047] In one possible implementation, the device further includes:

[0048] The third acquisition unit is used to acquire noise verification speech of different types and / or scenarios; and to acquire clean verification speech;

[0049] The second superposition unit is used to superimpose the noisy verification speech and the clean verification speech to obtain the verification speech, and to number the verification speech according to the classification of the noisy verification speech.

[0050] The third generation unit is used to generate the amplitude spectrum features of the verification speech;

[0051] The input unit is used to input the amplitude spectrum features of the verification speech into the speech denoising model to obtain the noisy speech prediction result and the clean speech prediction result in the verification speech;

[0052] The update unit is used to update the speech denoising model by reusing the verification speech as the sample speech when the clean speech prediction result of the verification speech is inconsistent with the clean verification speech.

[0053] In one possible implementation, the signal-to-noise ratio of the clean sample speech is not less than 25dB.

[0054] In one possible implementation, the noise reduction unit includes:

[0055] The first input subunit is used to input the amplitude spectrum features of the target speech into the coding layer of the speech denoising model to obtain a high-dimensional speech representation vector of the target speech;

[0056] The second input subunit is used to input the high-dimensional speech representation vector of the target speech into the convolutional layer of the speech denoising model to obtain the feature vector of the target speech under different types and / or scenarios.

[0057] The third input subunit is used to input the high-dimensional speech representation vector of the target speech and the feature vectors of the target speech under different types and / or scenarios into the attention layer of the speech denoising model for weighted processing to obtain the weighted speech representation vector.

[0058] The decoding subunit is used to input the weighted speech representation vector into the decoding layer of the speech denoising model for decoding processing to obtain the amplitude spectrum features of the noisy speech in the target speech, and to determine the amplitude spectrum features of the clean speech in the target speech using the amplitude spectrum features of the target speech and the amplitude spectrum features of the noisy speech, so as to identify the noisy speech and the clean speech in the target speech.

[0059] This application also provides a voice noise reduction device, including: a processor, a memory, and a system bus;

[0060] The processor and the memory are connected via the system bus;

[0061] The memory is used to store one or more programs, the one or more programs including instructions, which, when executed by the processor, cause the processor to perform any of the above-described implementations of the speech noise reduction method.

[0062] This application also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described speech noise reduction methods.

[0063] This application also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the above-described speech noise reduction methods.

[0064] This application provides a speech denoising method, apparatus, storage medium, and device. First, the target speech to be denoised is acquired, and then the amplitude spectrum features of the target speech are generated. Next, the amplitude spectrum features of the target speech can be input into a pre-constructed speech denoising model to identify noisy speech and clean speech in the target speech. The speech denoising model includes an encoding layer, a convolutional layer, an attention layer, and a decoding layer. The speech denoising model is trained using noisy speech data of different types and / or scenarios and a first objective function, a second objective function, and a third objective function.

[0065] As can be seen, this application first constructs a speech denoising model using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function. This allows the model to learn more directly the commonalities of the same type of noise and the differences between different types of noise, which is more conducive to balancing the denoising effect of the model under various noise types and / or scenarios. Then, by inputting the amplitude spectrum features of the target speech into the denoising model for denoising processing, the noise in the signal can be estimated more specifically, and then removed from the target speech to obtain the final clean speech, thereby effectively improving the denoising effect. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 A schematic flowchart of a speech noise reduction method provided in an embodiment of this application;

[0068] Figure 2 This is a schematic diagram of the structure of the speech denoising model provided in the embodiments of this application;

[0069] Figure 3 This is a schematic diagram of the composition of a speech noise reduction device provided in an embodiment of this application. Detailed Implementation

[0070] With the development of voice communication and human-computer interaction technologies, the application scenarios of these technologies, especially in noisy environments, are becoming increasingly complex, making the need for voice signal noise reduction increasingly urgent. Traditional voice noise reduction methods typically include the following two types:

[0071] I. Noise Reduction Methods Based on Traditional Statistical Signal Processing:

[0072] Traditional statistical signal processing methods rely on the fundamental assumption that noisy signals are superimposed from clean target signals and noise signals, and that the clean and target signals are statistically uncorrelated. However, because target signals are highly variable and difficult to predict, noise reduction can be achieved by directly subtracting the noise signals if the noise signal can be estimated reasonably well. These methods include spectral subtraction, MCRA, IMCRA, and LSA, all focusing on estimating the spectral characteristics of the noise. These methods are highly effective for stationary noise, such as white noise, and are controllable and stable. However, their noise reduction performance is significantly worse for non-stationary, abrupt noise.

[0073] As can be seen, statistical signal processing-based methods, which employ expert-designed noise estimation techniques, perform well for stationary noise, but are more challenging to estimate noise types with significant variations. Although the algorithms offer good controllability and stability, their performance in real-world applications is often lower than expected.

[0074] II. Supervised Regression Training Method Based on Neural Networks:

[0075] Neural network-based speech denoising methods involve establishing a network model structure and, through supervised training, allowing the model to gradually learn certain features of clean speech and noise in noisy speech. The clean speech features are then extracted and used as the network model's output. Many common model structures exist, such as the earliest deep neural networks (DNNs) and convolutional neural networks (CNNs), followed by sequence modeling-based networks like Long Short-Term Memory (LSTM) and GRU, and more recently, CRN and TCN. These networks are all seeking structures with better modeling capabilities to improve denoising performance. This denoising approach provides the network with more and broader coverage of noisy and clean data, allowing it to learn the features of noise and speech in noisy signals. By constraining the difference between the output signal and the target clean signal, the mean squared loss function (MSE Loss) is typically used for training.

[0076] As can be seen, this denoising method treats the denoising task as a regression problem. Regardless of changes in model structure, the ultimate goal remains the minimization of the second-order variance between the output estimated signal and the target signal, still involving signal-level information. Therefore, as models become increasingly complex, the denoising performance of this type of method becomes increasingly difficult to optimize and improve. That is, for different noise scenarios, the model can only extract features of different noises from the training data. For different noise types or scenarios, the model's learning level and the final denoising performance are difficult to control. Usually, the denoising performance in certain scenarios can only be improved by adjusting the data ratio of specific scenarios, which often leads to a deterioration in performance in other scenarios.

[0077] To address the aforementioned shortcomings, this application provides a speech denoising method. First, the target speech to be denoised is acquired, and then the amplitude spectrum features of the target speech are generated. Next, the amplitude spectrum features of the target speech can be input into a pre-constructed speech denoising model to identify noisy and clean speech in the target speech. The speech denoising model includes an encoding layer, a convolutional layer, an attention layer, and a decoding layer. The speech denoising model is trained using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function.

[0078] As can be seen, this application first constructs a speech denoising model using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function. This allows the model to learn more directly the commonalities of the same type of noise and the differences between different types of noise, which is more conducive to balancing the denoising effect of the model under various noise types and / or scenarios. Then, by inputting the amplitude spectrum features of the target speech into the denoising model for denoising processing, the noise in the signal can be estimated more specifically, and then removed from the target speech to obtain the final clean speech, thereby effectively improving the denoising effect.

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0080] First Embodiment

[0081] See Figure 1 This is a flowchart illustrating a speech noise reduction method provided in this embodiment. The method includes the following steps:

[0082] S101: Obtain the target speech to be denoised.

[0083] In this embodiment, any noisy speech that needs to be denoised (noise signal removed) is defined as the target speech. It should be noted that this embodiment does not limit the language type of the target speech; for example, the target speech can be composed of Chinese or English, etc. Similarly, this embodiment does not limit the length of the target speech; for example, the target speech can be a sentence or a paragraph, etc. Furthermore, this embodiment does not limit the acquisition scenario of the target speech; for example, the target speech can be acquired in different scenarios such as a street, a conference room, or a train station.

[0084] It is understood that the target speech can be obtained through recording or other means as needed. For example, it can be noisy speech recorded on the street or a recording in a conference room. At the same time as the target speech is obtained, the solution provided in this embodiment is used to perform noise reduction processing on the target speech in order to identify the noise signals and clean speech contained in the target speech.

[0085] S102: Generate amplitude spectrum features of the target speech.

[0086] In this embodiment, after obtaining the target speech to be denoised through step S101, in order to effectively improve the denoising effect, the amplitude spectrum feature of the target speech can be generated by using existing or future amplitude spectrum feature calculation methods, and the amplitude spectrum feature can be used as the basis for denoising, so as to achieve effective denoising of the target speech through subsequent step S103.

[0087] It should be noted that the embodiments of this application do not limit the method for generating amplitude spectrum features of the target speech, nor do they limit the specific generation process. Appropriate generation methods and corresponding amplitude spectrum feature generation operations can be selected according to the actual situation.

[0088] S103: Input the amplitude spectrum features of the target speech into the pre-built speech denoising model to identify noisy speech and clean speech in the target speech; wherein, the speech denoising model includes an encoding layer, a convolutional layer, an attention layer and a decoding layer; the speech denoising model is trained by using noisy speech data of different types and / or scenarios and the first objective function, the second objective function and the third objective function.

[0089] In this embodiment, after obtaining the amplitude spectrum features of the target speech through step S102, in order to effectively improve the noise reduction effect of the target speech, the amplitude spectrum features of the target speech can be further input into the pre-constructed speech noise reduction model to identify the noisy speech and clean speech contained in the target speech.

[0090] It should be noted that, in order to improve the noise reduction effect on the target speech, this application pre-constructs a speech noise reduction model, such as... Figure 2 As shown, the model includes an encoding layer (such as...). Figure 2 Encoder in the middle), convolutional layer (such as...) Figure 2 Conv1, Conv1…ConvN-1), attention layers (such as...) Figure 2 Attention in (and decoding layer (e.g.)) and decoding layer (e.g.) Figure 2 The denoising model is trained using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function. By inputting the amplitude spectrum features of the target speech into this denoising model for noise reduction, noise in the signal can be estimated more effectively and then removed from the target speech, resulting in clean speech and thus significantly improving the denoising performance.

[0091] Specifically, one possible implementation is that the process of "inputting the amplitude spectrum features of the target speech into a pre-built speech denoising model to identify the noisy speech and clean speech in the target speech" in step S103 may include the following steps S1031-S1034:

[0092] Step S1031: Input the amplitude spectrum features of the target speech into the coding layer of the speech denoising model to obtain the high-dimensional speech representation vector of the target speech.

[0093] In this implementation, after generating the amplitude spectrum features of the target speech in step S102, the amplitude spectrum features of the target speech can be further input into the encoder layer of the speech denoising model to obtain a high-dimensional speech representation vector of the target speech, which is then used to execute the subsequent step S1032.

[0094] Step S1032: Input the high-dimensional speech representation vector of the target speech into the convolutional layer of the speech denoising model to obtain the feature vector of the target speech under different types and / or scenarios.

[0095] In this implementation, after inputting the amplitude spectrum features of the target speech into the coding layer of the speech denoising model in step S1031 to obtain the high-dimensional speech representation vector of the target speech, the high-dimensional speech representation vector of the target speech can be further input into the convolutional layer of the speech denoising model to obtain the feature vectors of the target speech under different types and / or scenarios, which are then used to execute the subsequent step S1033.

[0096] Step S1033: Input the high-dimensional speech representation vector of the target speech and the feature vectors of the target speech under different types and / or scenarios into the attention layer of the speech denoising model for weighted processing to obtain the weighted speech representation vector.

[0097] In this implementation, after obtaining the high-dimensional speech representation vector of the target speech in step S1031 and the feature vectors of the target speech under different types and / or scenarios in step S1032, the high-dimensional speech representation vector of the target speech and the feature vectors of the target speech under different types and / or scenarios can be further input into the attention layer of the speech denoising model for weighted processing to achieve feature enhancement, and the weighted speech representation vector is obtained for executing the subsequent step S1034.

[0098] Step S1034: Input the weighted speech representation vector into the decoding layer of the speech denoising model for decoding processing to obtain the amplitude spectrum features of the noisy speech in the target speech. Then, using the amplitude spectrum features of the target speech and the amplitude spectrum features of the noisy speech, determine the amplitude spectrum features of the clean speech in the target speech, thereby identifying the noisy speech and the clean speech in the target speech.

[0099] In this implementation, after obtaining the weighted speech representation vector through step S1033, the weighted speech representation vector can be further input into the decoding layer of the speech denoising model for decoding processing to obtain the amplitude spectrum feature (Noise Spec(mag)) of the noisy speech in the target speech. Then, using the amplitude spectrum feature of the noisy speech, the pure noise signal in the target speech can be predicted. Then, by subtracting the amplitude spectrum feature of the noisy speech from the amplitude spectrum feature of the target speech, the amplitude spectrum feature of the clean speech in the target speech can be determined. Thus, the clean speech signal contained in the target speech can be predicted by using the amplitude spectrum feature of the clean speech.

[0100] Next, this embodiment will introduce the construction process of the speech denoising model. Figure 2 The diagram illustrates the structure and construction process of the speech denoising model provided in this embodiment. The specific training process may include the following steps A1-A4:

[0101] Step A1: Obtain noisy speech samples of different types and / or scenarios; and obtain clean speech samples.

[0102] In this embodiment, extensive preparatory work is required to construct the speech denoising model. First, a large amount of noise speech data of different types and / or scenarios needs to be collected. For example, different types of noise speech data such as white noise, colored noise, non-stationary noise, and impulse noise, or noise speech data of different scenarios such as wind noise, kitchen noise, conference room noise, street noise, and station noise, ensuring that this speech data covers more scenarios or noise types. Then, the noise speech data is numbered according to its type or scenario, denoted as Noise_ID. The specific numbering method is not limited; for example, numbers 1, 2, etc., can be used. Additionally, a large amount of clean speech data with a signal-to-noise ratio of at least 25dB needs to be collected through a recording studio or other open-source channels. Each collected piece of noise speech data can then be used as a noise sample speech, and each collected piece of clean speech data can be used as a clean sample speech for subsequent step A2.

[0103] Step A2: Superimpose the noisy speech sample and the clean speech sample to obtain the sample speech, and number the sample speech according to the classification of the noisy speech sample.

[0104] It should be noted that, to improve the model's noise reduction effect, after obtaining a large number of noise sample speech of different types and / or scenarios in step A1, and a large number of clean sample speech speech, the noise sample speech and clean sample speech can be further processed to unify their formats. For example, both the noise sample speech and clean sample speech can be converted into single-channel pulse (PCM) data with a sampling rate of 16kHz and float32 quantization (signal range between ±1), which can be used as the preprocessed noise sample speech and clean sample speech. Then, the obtained preprocessed noise sample speech and clean sample speech can be superimposed to obtain sample speech, which is used to perform the subsequent step A3, and the sample speech can be numbered according to the classification of the noise sample speech.

[0105] Specifically, after obtaining the noisy and clean speech samples, the process begins by iterating through all the clean speech samples K times (K being a positive integer greater than 0). Each time, one clean speech sample is randomly or sequentially selected, denoted as x(t). For example, assuming a total of N (K being a positive integer greater than 0) clean speech samples are collected, repeating the iteration K times will extract a total of K*N clean speech samples. Then, one noisy speech sample is randomly selected from all the noisy speech samples, or a segment of noisy speech with the same length as the currently extracted clean speech sample is randomly selected, denoted as n(t). For example, if the original noisy speech sample data is longer than the clean speech sample data and does not meet the length requirement, the noisy speech sample data can be repeated several times to make its length exceed the length of the clean speech sample data, and then a segment of noisy speech with the same length as the currently extracted clean speech sample data can be randomly selected.

[0106] Next, the signals can be superimposed according to the following formula, based on a uniform random distribution and a preset signal-to-noise ratio (defined as α) and signal energy (defined as β), to obtain the sample speech y′(t), while simultaneously saving the proportionally scaled noise signal n′(t) and the clean speech signal x′(t):

[0107]

[0108]

[0109]

[0110] y(t)=x(t)+10 c / 10 *n(t)

[0111] σ y =max(|y(t)| 2 )

[0112]

[0113]

[0114]

[0115] Where, σ x σ represents the total energy of a clean sample speech x(t); n σ represents the total energy of the noise sample speech n(t); c represents the scaling factor of the noise sample speech n(t) so that the signal-to-noise ratio and signal energy of the intermediate sample speech y(t) can meet the preset signal-to-noise ratio α and signal energy β. The specific values ​​of the preset signal-to-noise ratio α and signal energy β are not limited; for example, the preset signal-to-noise ratio α can be set to 10dB between 0 and 15dB, and the signal energy β can be set to 0.5 or higher than 0. y y'(t) represents the instantaneous energy of the intermediate sample speech y(t) with the maximum time; y'(t) represents the generated sample speech; x'(t) represents the clean speech contained in the sample speech; n'(t) represents the noisy sample speech contained in the sample speech.

[0116] Furthermore, the ID of the sample speech can be consistent with the noise sample speech n′(t) it contains, i.e., Noise_ID. Thus, each sample speech y′(t) generated by the above formula has a corresponding clean speech x′(t), a noise sample speech n′(t), and a noise ID Noise_ID. Ultimately, this ensures that a set of training data consists of y′(t), x′(t), and Noise_ID.

[0117] In this way, the final K*N sets of training data can be randomly shuffled and saved. In each set of training data containing sample speech, y′(t) is used to generate the Noisy Magnitude Spec, denoted as |Y′||; x′(t) is used to generate the Clean Magnitude Spec, denoted as |X′|; and Noise_ID is used to generate the Noise One-hot vector (i.e., an N-dimensional vector, where the dimension corresponding to Noise_ID is 1 and the other dimensions are 0, and N represents the total number of categories or scenes to which the noise sample speech belongs), denoted as V.

[0118] Step A3: Generate amplitude spectrum features of the sample speech.

[0119] After obtaining the sample speech in step A2, a method similar to generating the amplitude spectrum features of the target speech in step S102 can be used to replace the target speech with the sample speech, thereby generating the amplitude spectrum features of each sample speech (e.g., Figure 2The Noisy Spec(mag) shown here is related to the description of step S102 and will not be repeated here. It is used to perform the subsequent step A4.

[0120] Step A4: Train the initial speech denoising model using the amplitude spectrum features of the sample speech, the first objective function, the second objective function, the third objective function, and the clean sample speech to generate a speech denoising model.

[0121] It should be noted that, in order to improve the denoising effect of the speech denoising model, the initial speech denoising model used in this application includes an encoder layer, a decoder layer, N convolutional layers, and an attention layer, as follows: Figure 2 As shown in the figure. Here, N is the total number of categories or scenes of the noise sample speech mentioned above.

[0122] During model training, after generating the amplitude spectrum features of the sample speech in step A3, the initial speech denoising model can be further trained using these features, the first objective function, the second objective function, the third objective function, and clean sample speech. During training, the model parameters can be continuously updated based on the changes in the function values ​​of the first, second, and third objective functions until these functions meet the requirements, such as reaching a minimum value with minimal change (basically unchanged), or reaching a preset maximum number of iterations (e.g., 100). At this point, updating the model parameters stops, completing the training of the speech denoising model and generating a trained speech denoising model.

[0123] The first objective function can be, but is not limited to, the mean squared loss function (MSE Loss), used to reduce the error between the clean speech output by the speech denoising model and the clean sample speech, so that the error between the amplitude of the final clean signal output by the model and the amplitude spectrum of the real clean sample speech signal becomes smaller and smaller. The specific calculation formula of the first objective function is as follows:

[0124]

[0125] Among them, L MSE Denotes the first objective function; |X′| represents the amplitude spectrum of the clean speech signal output by the model; |X′| represents the amplitude spectrum of the clean sample speech signal.

[0126] The second objective function can be, but is not limited to, the cosine similarity loss function (CS Loss), which is applied to the decoding layer of the denoising model, such as... Figure 2 As shown, this is used to reduce the similarity of speech samples from different types and / or scenarios during training, as well as the similarity of speech samples from the same type and / or scenario, and to optimize each speech sample in a direction where channel information cannot be distinguished. For ease of explanation, the estimated noise amplitude spectrum output by the model's decoding layer is denoted as... Because a batch of training data can contain multiple sentences (i.e., multiple sample speech), different sentences are distinguished by subscripts as follows:

[0127]

[0128] Thus, the specific formula for calculating the second objective function is as follows:

[0129]

[0130] Among them, L CS The second objective function is represented by M; M represents the total number of constraint processing operations performed on each pair of noise sample speech in a batch of training data. For example, if a batch of training data contains 10 noise sample speech, then the value of M is 45.

[0131] The third objective function can be, but is not limited to, the mean absolute error loss function (L1 Loss), and is applied to the attention layer of the denoising model, such as... Figure 2 As shown, this is used to assign weighted coefficients generated by the attention layer to the correct output features of the convolutional layer during training. For example, assuming the Noise_ID of the currently trained sentence is 2, then the generated attention coefficients (defined here as...) It should approximate the true value V = [0, 0, 1, ..., 0]. T The specific formula for calculating the third objective function is as follows:

[0132]

[0133] Among them, L L1 This represents the third objective function; The generated Attention coefficients are shown in the following formula:

[0134] h K =W K *h_enc

[0135] h Q (i)=W Q *h_conv(i)

[0136]

[0137] Among them, W K W Q H represents the random matrix in the attention layer; h_enc represents the high-dimensional sample speech representation vector output by the coding layer; h_conv(i) represents the feature vector output by the i-th convolutional (Conv) layer; H Q T Represents N h Q (i) is the concatenation value.

[0138] In the model's attention layer, through The formula for weighted summation of the parallel Conv layers is as follows:

[0139] h V (i)=W V *h_conv(i)

[0140]

[0141] In this way, by inputting the weighted sample speech representation vector (Attention out) output by the model's attention layer into the model's decoder layer, the noise reduction result can be obtained.

[0142] It should be noted that in actual training, the first objective function (L) can also be used. MSE ), second objective function (L CS ), third objective function (L L1 Jointly training the network with the overall objective function (defined here as L) can effectively update the network parameters, resulting in better noise reduction performance. The proportions of the overall objective function are not limited and can be set based on actual conditions and experience. For example, the formula for calculating the overall objective function L is as follows:

[0143] L = L MSE +a*L CS +b*L L1

[0144] The values ​​of a and b can be obtained through experiments, and this application does not impose any restrictions. For example, a can be set to 1 and b can be set to 0.2, etc.

[0145] Therefore, during the training process, the model parameters of the denoising model can be continuously updated according to the changes in the function value of the total objective function L, until the function value of the total objective function L meets the requirements, such as reaching the minimum value and the change is very small (basically unchanged). Then, the update of the model parameters is stopped, the training of the speech denoising model is completed, and the trained speech denoising model is generated.

[0146] Thus, in this embodiment, by introducing the labels of noise from different scenarios (or types) into the training loss constraint, the model can more directly learn the commonalities of the same noise and the differences between different noises. This is more conducive to balancing the noise reduction effect of the model under various noise scenarios (or types). Furthermore, in the model structure, a dedicated Conv layer is adapted for different noise scenarios (or types) to learn the features of a specific type of noise. Then, the Attention mechanism is used to determine the proportion and type of noise contained in the current noisy signal, so that the noise in the signal can be estimated more specifically. Then, the noise is removed from the original noisy signal to obtain the final clean speech signal.

[0147] Building upon this, after training and generating a speech denoising model based on sample speech data, the generated speech denoising model can be further validated using validation speech data. The specific validation process may include the following steps B1-B5:

[0148] Step B1: Obtain noisy verification speech of different types and / or scenarios; and obtain clean verification speech.

[0149] In this embodiment, in order to verify the speech denoising model, it is first necessary to acquire a large number of noise verification speech of different types and / or scenarios; and acquire clean verification speech for subsequent step B2.

[0150] Step B2: Superimpose the noisy verification speech and the clean verification speech to obtain the verification speech, and number the verification speech according to the classification of the noisy verification speech.

[0151] In this embodiment, noise verification speech of different types and / or scenarios is obtained through step B1; and after obtaining clean verification speech, a method similar to that used in step A2 to generate sample speech can be used to replace the noise sample speech, clean sample speech, and sample speech with noise verification speech, clean verification speech, and verification speech, respectively. For relevant details, please refer to the description of step A2, which will not be repeated here, in order to execute the subsequent step B3.

[0152] Step B3: Generate amplitude spectrum features of the verification speech.

[0153] In this embodiment, after obtaining the verification speech in step B2, a method similar to that used in step S102 to generate the amplitude spectrum features of the target speech can be adopted to replace the target speech with the verification speech, thereby generating the amplitude spectrum features of each verification speech. For details, please refer to the description of step S102, which will not be repeated here, and will be used to execute the subsequent step B4.

[0154] Step B4: Input the amplitude spectrum features of the validation speech into the speech denoising model to obtain the noisy speech prediction results and clean speech prediction results in the validation speech.

[0155] After generating the amplitude spectrum features of the verification speech in step B3, the amplitude spectrum features of the verification speech can be further input into the speech denoising model to obtain the noisy speech prediction results and clean speech prediction results in the verification speech, which can then be used to execute the subsequent step B5.

[0156] Step B5: When the clean speech prediction result of the verification speech is inconsistent with the clean verification speech, the verification speech is used again as the sample speech to update the speech denoising model.

[0157] After obtaining the clean speech prediction result in the verification speech through step B4, if the clean speech prediction result is inconsistent with the clean verification speech, the verification speech can be used again as the sample speech to update the parameters of the speech denoising model.

[0158] Through the above embodiments, the speech denoising model can be effectively validated using validation speech data. When the clean speech prediction result of the validation speech is inconsistent with the clean validation speech, the speech denoising model can be adjusted and updated in a timely manner, thereby helping to improve the denoising accuracy and precision of the speech denoising model.

[0159] In summary, the speech denoising method provided in this embodiment first obtains the target speech to be denoised, and then generates the amplitude spectrum features of the target speech. Next, the amplitude spectrum features of the target speech can be input into a pre-constructed speech denoising model to identify noisy speech and clean speech in the target speech. The speech denoising model includes an encoding layer, a convolutional layer, an attention layer, and a decoding layer. The speech denoising model is trained using noisy speech data of different types and / or scenarios and a first objective function, a second objective function, and a third objective function.

[0160] As can be seen, this application first constructs a speech denoising model using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function. This allows the model to learn more directly the commonalities of the same type of noise and the differences between different types of noise, which is more conducive to balancing the denoising effect of the model under various noise types and / or scenarios. Then, by inputting the amplitude spectrum features of the target speech into the denoising model for denoising processing, the noise in the signal can be estimated more specifically, and then removed from the target speech to obtain the final clean speech, thereby effectively improving the denoising effect.

[0161] Second Embodiment

[0162] This embodiment will introduce a speech noise reduction device; please refer to the above method embodiment for related content.

[0163] See Figure 3 This is a schematic diagram of the composition of a speech noise reduction device provided in this embodiment. The device 300 includes:

[0164] The first acquisition unit 301 is used to acquire the target speech to be denoised.

[0165] The first generation unit 302 is used to generate the amplitude spectrum features of the target speech;

[0166] The noise reduction unit 303 is used to input the amplitude spectrum features of the target speech into a pre-built speech noise reduction model to identify the noisy speech and clean speech in the target speech;

[0167] The speech denoising model includes an encoding layer, a convolutional layer, an attention layer, and a decoding layer; the speech denoising model is trained using noisy speech data of different types and / or scenarios, along with a first objective function, a second objective function, and a third objective function.

[0168] In one implementation of this embodiment, the apparatus further includes:

[0169] The second acquisition unit is used to acquire noisy speech samples of different types and / or scenarios; and to acquire clean speech samples.

[0170] The first superposition unit is used to superimpose the noise sample speech and the clean sample speech to obtain sample speech, and to number the sample speech according to the classification of the noise sample speech;

[0171] The second generation unit is used to generate the amplitude spectrum features of the sample speech;

[0172] The training unit is used to train the initial speech denoising model using the amplitude spectrum features of the sample speech, the first objective function, the second objective function, the third objective function, and the clean sample speech, to generate a speech denoising model.

[0173] In one implementation of this embodiment, the first objective function is a mean squared loss function; the mean squared loss function reduces the error between the clean speech output by the speech denoising model and the clean sample speech; the second objective function is a cosine similarity loss function; the cosine similarity loss function is applied to the decoding layer to reduce the similarity between noise sample speech of different types and / or scenarios, the similarity between noise sample speech of the same type and / or scenario, and to optimize each sample speech towards a direction where channel information cannot be distinguished; the third objective function is a mean absolute error loss function; the mean absolute error loss function is applied to the attention layer to allocate the weighting coefficients generated by the attention layer to the correct output features of the convolutional layer.

[0174] In one implementation of this embodiment, the apparatus further includes:

[0175] The preprocessing unit is used to perform format unification processing on the noise sample speech and the clean sample speech to obtain preprocessed noise sample speech and clean sample speech.

[0176] The first superposition unit is specifically used for:

[0177] The preprocessed noisy speech sample and the clean speech sample are superimposed to obtain the sample speech.

[0178] In one implementation of this embodiment, the apparatus further includes:

[0179] The third acquisition unit is used to acquire noise verification speech of different types and / or scenarios; and to acquire clean verification speech;

[0180] The second superposition unit is used to superimpose the noisy verification speech and the clean verification speech to obtain the verification speech, and to number the verification speech according to the classification of the noisy verification speech.

[0181] The third generation unit is used to generate the amplitude spectrum features of the verification speech;

[0182] The input unit is used to input the amplitude spectrum features of the verification speech into the speech denoising model to obtain the noisy speech prediction result and the clean speech prediction result in the verification speech;

[0183] The update unit is used to update the speech denoising model by reusing the verification speech as the sample speech when the clean speech prediction result of the verification speech is inconsistent with the clean verification speech.

[0184] In one implementation of this embodiment, the signal-to-noise ratio of the clean sample speech is not less than 25dB.

[0185] In one implementation of this embodiment, the noise reduction unit 303 includes:

[0186] The first input subunit is used to input the amplitude spectrum features of the target speech into the coding layer of the speech denoising model to obtain a high-dimensional speech representation vector of the target speech;

[0187] The second input subunit is used to input the high-dimensional speech representation vector of the target speech into the convolutional layer of the speech denoising model to obtain the feature vector of the target speech under different types and / or scenarios.

[0188] The third input subunit is used to input the high-dimensional speech representation vector of the target speech and the feature vectors of the target speech under different types and / or scenarios into the attention layer of the speech denoising model for weighted processing to obtain the weighted speech representation vector.

[0189] The decoding subunit is used to input the weighted speech representation vector into the decoding layer of the speech denoising model for decoding processing to obtain the amplitude spectrum features of the noisy speech in the target speech, and to determine the amplitude spectrum features of the clean speech in the target speech using the amplitude spectrum features of the target speech and the amplitude spectrum features of the noisy speech, so as to identify the noisy speech and the clean speech in the target speech.

[0190] Furthermore, embodiments of this application also provide a voice noise reduction device, including: a processor, a memory, and a system bus;

[0191] The processor and the memory are connected via the system bus;

[0192] The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any of the above-described implementations of the speech noise reduction method.

[0193] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described implementations of the speech noise reduction method.

[0194] Furthermore, this application also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the above-described speech noise reduction methods.

[0195] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0196] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0197] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0198] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech noise reduction method, characterized in that, include: Obtain the target speech to be denoised; Generate the amplitude spectrum features of the target speech; The amplitude spectrum features of the target speech are input into a pre-built speech denoising model to identify noisy speech and clean speech in the target speech; The speech denoising model includes an encoding layer, a convolutional layer, an attention layer, and a decoding layer; the speech denoising model is trained using noisy speech data of different types and / or scenarios and a first objective function, a second objective function, and a third objective function. The speech denoising model is constructed as follows: Acquire noisy speech samples of different types and / or scenarios; and acquire clean speech samples; The noisy speech sample and the clean speech sample are superimposed to obtain the sample speech, and the sample speech is numbered according to the classification of the noisy speech sample. Generate the amplitude spectrum features of the sample speech; An initial speech denoising model is trained using the amplitude spectrum features of the sample speech, a first objective function, a second objective function, a third objective function, and the clean sample speech to generate a speech denoising model. The first objective function is a mean square loss function, which reduces the error between the clean speech output by the speech denoising model and the clean sample speech. The second objective function is a cosine similarity loss function, which is applied to the decoding layer to reduce the similarity between noise sample speech of different types and / or scenarios, and the similarity between noise sample speech of the same type and / or scenario, and to optimize each sample speech towards directions where channel information cannot be distinguished. The third objective function is a mean absolute error loss function, which is applied to the attention layer to distribute the weighting coefficients generated by the attention layer to the correct output features of the convolutional layer.

2. The method according to claim 1, characterized in that, Before performing the superposition process on the noisy sample speech and the clean sample speech to obtain the sample speech, the method further includes: The noise sample speech and the clean sample speech are subjected to format unification processing to obtain preprocessed noise sample speech and clean sample speech; The process of superimposing the noisy sample speech and the clean sample speech to obtain sample speech includes: The preprocessed noisy speech sample and the clean speech sample are superimposed to obtain the sample speech.

3. The method according to claim 1, characterized in that, The method further includes: Acquire noisy verification speech of different types and / or scenarios; and acquire clean verification speech; The noisy verification speech and the clean verification speech are superimposed to obtain the verification speech, and the verification speech is numbered according to the classification of the noisy verification speech. Generate the amplitude spectrum features of the verification speech; The amplitude spectrum features of the verification speech are input into the speech denoising model to obtain the noisy speech prediction results and clean speech prediction results in the verification speech; When the clean speech prediction result of the verification speech is inconsistent with the clean verification speech, the verification speech is used again as the sample speech to update the speech denoising model.

4. The method according to any one of claims 1-3, characterized in that, The signal-to-noise ratio of the clean sample speech is not less than 25dB.

5. The method according to claim 1, characterized in that, The step of inputting the amplitude spectrum features of the target speech into a pre-built speech denoising model to identify noisy and clean speech in the target speech includes: The amplitude spectrum features of the target speech are input into the coding layer of the speech denoising model to obtain a high-dimensional speech representation vector of the target speech; The high-dimensional speech representation vector of the target speech is input into the convolutional layer of the speech denoising model to obtain the feature vectors of the target speech under different types and / or scenarios. The high-dimensional speech representation vector of the target speech and the feature vectors of the target speech under different types and / or scenarios are input into the attention layer of the speech denoising model for weighted processing to obtain the weighted speech representation vector. The weighted speech representation vector is input into the decoding layer of the speech denoising model for decoding processing to obtain the amplitude spectrum features of the noisy speech in the target speech. Then, using the amplitude spectrum features of the target speech and the amplitude spectrum features of the noisy speech, the amplitude spectrum features of the clean speech in the target speech are determined, thereby identifying the noisy speech and the clean speech in the target speech.

6. A voice noise reduction device, characterized in that, include: The first acquisition unit is used to acquire the target speech to be denoised. The first generation unit is used to generate the amplitude spectrum features of the target speech; The noise reduction unit is used to input the amplitude spectrum features of the target speech into a pre-built speech noise reduction model to identify the noisy speech and clean speech in the target speech; The speech denoising model includes an encoding layer, a convolutional layer, an attention layer, and a decoding layer; the speech denoising model is trained using noisy speech data of different types and / or scenarios and a first objective function, a second objective function, and a third objective function. The device further includes: The second acquisition unit is used to acquire noisy speech samples of different types and / or scenarios; and to acquire clean speech samples. The first superposition unit is used to superimpose the noise sample speech and the clean sample speech to obtain sample speech, and to number the sample speech according to the classification of the noise sample speech; The second generation unit is used to generate the amplitude spectrum features of the sample speech; The training unit is used to train an initial speech denoising model using the amplitude spectrum features of the sample speech, a first objective function, a second objective function, a third objective function, and the clean sample speech, to generate a speech denoising model. The first objective function is a mean square loss function, which reduces the error between the clean speech output by the speech denoising model and the clean sample speech. The second objective function is a cosine similarity loss function, which is applied to the decoding layer to reduce the similarity between noise sample speech of different types and / or scenarios, and the similarity between noise sample speech of the same type and / or scenario, and to optimize each sample speech towards a direction where channel information cannot be distinguished. The third objective function is a mean absolute error loss function, which is applied to the attention layer to allocate the weighting coefficients generated by the attention layer to the correct output features of the convolutional layer.

7. A voice noise reduction device, characterized in that, include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any one of claims 1-5.

Images