Speech enhancement model training method and device and speech enhancement method and device

Abstract

The present disclosure provides a training method and device for a speech enhancement model and a speech enhancement method and device. The training method may include the following steps: obtaining training samples including clean speech signals and noisy speech signals; inputting information of the noisy speech signals into a feature extraction network to obtain features corresponding to the noisy speech signals; inputting features into a denoising network to obtain prediction information predicted by the denoising network; inputting features into a speed-shifting and pitch-shifting determination network to obtain a speed-shifting and pitch-shifting estimation value of the noisy speech signals predicted by the speed-shifting and pitch-shifting determination network; adjusting parameters of the feature extraction network, the denoising network, and the speed-shifting and pitch-shifting determination network based on information of the noisy speech signals, prediction information, speed-shifting and pitch-shifting estimation values, and information of the clean speech signals to train the speech enhancement model.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and speech processing, and in particular to a method and device for training a speech enhancement model for performing speech enhancement on input speech, and a speech enhancement method and device. Background Art

[0002] With the rapid development of deep learning technology, speech enhancement technology based on neural networks has also been continuously improving in recent years. Through large-scale data training, current neural networks can effectively extract clean speech signals from noisy speech signals.

[0003] Short videos have become the main way for people to obtain information and have fun in today's society. In the process of short video content production, in order to improve the diversity and playability of the content, content producers often process the sound with variable speed and pitch. However, the related technology does not have an ideal effect on the enhancement of the sound processed with variable speed and pitch. Therefore, a solution for robustly enhancing the sound with variable speed and pitch is urgently needed. Summary of the invention

[0004] The present disclosure provides a method and device for training a speech enhancement model for performing speech enhancement on input speech, as well as a speech enhancement method and device, an electronic device and a storage medium, so as to at least solve the robustness problem of speech enhancement of speed-varied and pitch-varied audio by speech enhancement technology based on neural networks.

[0005] According to a first aspect of an embodiment of the present disclosure, a training method for a speech enhancement model is provided, the speech enhancement model comprising a feature extraction network and a noise reduction network, and the training method may include the following steps: obtaining training samples, wherein the training samples comprise a clean speech signal and a noisy speech signal, the clean speech signal being a noise-free speech signal that has not been subjected to speed change and pitch change processing, and the noisy speech signal being obtained by subjecting the clean speech signal to speed change and pitch change processing and noise mixing processing; inputting information of the noisy speech signal into the feature extraction network to obtain features corresponding to the noisy speech signal; inputting the features into the noise reduction network to obtain prediction information predicted by the noise reduction network, wherein the prediction information is used to generate an enhanced speech signal of the noisy speech signal; inputting the features into a speed change and pitch change determination network to obtain a speed change and pitch change estimation value of the noisy speech signal predicted by the speed change and pitch change determination network; adjusting parameters of the feature extraction network, the noise reduction network and the speed change and pitch change determination network based on information of the noisy speech signal, the prediction information, the speed change and pitch change estimation value and information of the clean speech signal to train the speech enhancement model.

[0006] Optionally, adjusting the parameters of the feature extraction network, the denoising network and the speed change and pitch change determination network based on the information of the noisy speech signal, the prediction information, the speed change and pitch change estimation value and the information of the clean speech signal to train the speech enhancement model may include: calculating a first loss function based on the information of the noisy speech signal, the prediction information and the information of the clean speech signal; calculating a second loss function based on the speed change and pitch change estimation value and the actual value of the speed change and pitch change when the speed change and pitch change processing is performed on the clean speech signal; and adjusting the parameters of the feature extraction network, the denoising network and the speed change and pitch change determination network based on the first loss function and the second loss function.

[0007] Optionally, the information of the noisy speech signal may include an amplitude spectrum of the noisy speech signal, and the prediction information may include a masking matrix for obtaining an enhanced speech signal, wherein calculating the first loss function based on the information of the noisy speech signal, the prediction information and the information of the clean speech signal may include: multiplying the masking matrix for obtaining the enhanced speech signal and the amplitude spectrum of the noisy speech signal to obtain the amplitude spectrum of the enhanced speech signal; and calculating the first loss function based on the amplitude spectrum of the enhanced speech signal and the amplitude spectrum of the clean speech signal.

[0008] Optionally, calculating the second loss function based on the speed change and pitch change estimation value and the speed change and pitch change actual value when performing speed change and pitch change processing on the clean speech signal may include: calculating the first error function based on the speed change estimation value in the speed change and pitch change actual value in the speed change and pitch change actual value; calculating the second error function based on the pitch change estimation value in the speed change and pitch change estimation value and the pitch change actual value in the speed change and pitch change actual value; calculating the second loss function based on the first error function and the second error function.

[0009] Optionally, calculating the second loss function based on the speed change and pitch change estimation value and the speed change and pitch change actual value when performing speed change and pitch change processing on the clean speech signal may include: adjusting the speed change and pitch change estimation value and the speed change and pitch change actual value to values ​​within a preset range; and calculating the second loss function based on the error between the adjusted speed change and pitch change estimation value and the adjusted speed change and pitch change actual value.

[0010] Optionally, adjusting the speed change and pitch change estimation value and the speed change and pitch change actual value to values ​​within a preset range may include: obtaining the maximum faster speed, the maximum slower speed, the maximum up-modulation speed and the maximum down-modulation speed when performing speed change and pitch change processing on the clean speech signal; based on the maximum faster speed, the maximum slower speed and a preset speed change value adjustment strategy, adjusting the speed change estimation value in the speed change and pitch change estimation value and the speed change actual value in the speed change and pitch change actual value to values ​​within a preset range; based on the maximum up-modulation speed, the maximum down-modulation speed and a preset pitch adjustment strategy, adjusting the pitch change estimation value in the speed change and pitch change estimation value and the pitch change actual value in the speed change and pitch change actual value to values ​​within a preset range.

[0011] Optionally, calculating the second loss function based on the speed change pitch estimation value and the speed change pitch actual value when performing speed change pitch processing on the clean speech signal may include: using the speed change pitch estimation value and the speed change pitch actual value for the speech frame to calculate the second loss function.

[0012] Optionally, the information of the noisy speech signal may include frequency domain information of the noisy speech signal, and the prediction information may include frequency domain information for obtaining an enhanced speech signal; or, the information of the noisy speech signal may include time domain information of the noisy speech signal, and the prediction information may include time domain information for obtaining an enhanced speech signal.

[0013] According to a second aspect of an embodiment of the present disclosure, a speech enhancement method is provided, which may include: obtaining input speech; inputting the input speech into a speech enhancement model to obtain enhanced speech corresponding to the input speech, wherein the speech enhancement model is obtained based on the training method as described above.

[0014] According to a third aspect of an embodiment of the present disclosure, a training device for a speech enhancement model is provided, wherein the speech enhancement model includes a feature extraction network and a noise reduction network, and the training device may include: an acquisition module, configured to acquire training samples, wherein the training samples include a clean speech signal and a noisy speech signal, the clean speech signal is a noise-free speech signal that has not been subjected to speed change and pitch change processing, and the noisy speech signal is obtained by performing speed change and pitch change processing and noise mixing processing on the clean speech signal; a feature extraction module, configured to input information of the noisy speech signal into the feature extraction network to obtain features corresponding to the noisy speech signal; and a noise reduction module. A block is configured to input the feature into the denoising network to obtain prediction information predicted by the denoising network, wherein the prediction information is used to generate an enhanced speech signal of the noisy speech signal; a speed change and pitch change determination module is configured to input the feature into the speed change and pitch change determination network to obtain a speed change and pitch change estimation value of the noisy speech signal predicted by the speed change and pitch change determination network; a training module is configured to adjust the parameters of the feature extraction network, the denoising network and the speed change and pitch change determination network based on the information of the noisy speech signal, the prediction information, the speed change and pitch change estimation value and the information of the clean speech signal to train the speech enhancement model.

[0015] Optionally, the training module can be configured to: calculate a first loss function based on the information of the noisy speech signal, the prediction information and the information of the clean speech signal; calculate a second loss function based on the speed change and pitch change estimation value and the actual value of the speed change and pitch change when the speed change and pitch change processing is performed on the clean speech signal; and adjust the parameters of the feature extraction network, the denoising network and the speed change and pitch change determination network according to the first loss function and the second loss function.

[0016] Optionally, the information of the noisy speech signal may include an amplitude spectrum of the noisy speech signal, and the prediction information may include a masking matrix for obtaining an enhanced speech signal, wherein the training module may be configured to: multiply the masking matrix for obtaining the enhanced speech signal and the amplitude spectrum of the noisy speech signal to obtain the amplitude spectrum of the enhanced speech signal; and calculate the first loss function based on the amplitude spectrum of the enhanced speech signal and the amplitude spectrum of the clean speech signal.

[0017] Optionally, the training module can be configured to: calculate a first error function based on the speed shift estimated value in the speed shift pitch estimated value and the speed shift actual value in the speed shift pitch actual value; calculate a second error function based on the pitch estimated value in the speed shift pitch estimated value and the pitch actual value in the speed shift pitch actual value; calculate the second loss function based on the first error function and the second error function.

[0018] Optionally, the training module may be configured to: adjust the speed change modulation estimate value and the speed change modulation actual value to values ​​within a preset range; and calculate a second loss function based on an error between the adjusted speed change modulation estimate value and the adjusted speed change modulation actual value.

[0019] Optionally, the training module can be configured to: obtain the maximum faster speed, the maximum slower speed, the maximum up-modulation speed and the maximum down-modulation speed when performing speed and pitch changing processing on the clean speech signal; based on the maximum faster speed, the maximum slow speed and a preset speed value adjustment strategy, adjust the speed estimated value in the speed and pitch changing estimated value and the speed actual value in the speed and pitch changing actual value to a value within a preset range; based on the maximum up-modulation speed, the maximum down-modulation speed and a preset pitch adjustment strategy, adjust the pitch estimated value in the speed and pitch changing estimated value and the pitch actual value in the speed and pitch changing actual value to a value within a preset range.

[0020] Optionally, the training module may be configured to calculate the second loss function using the speed change and pitch change estimation value and the speed change and pitch change actual value for the speech frame.

[0021] Optionally, the information of the noisy speech signal may include frequency domain information of the noisy speech signal, and the prediction information may include frequency domain information for obtaining an enhanced speech signal; or, the information of the noisy speech signal may include time domain information of the noisy speech signal, and the prediction information may include time domain information for obtaining an enhanced speech signal.

[0022] According to the fourth aspect of an embodiment of the present disclosure, a speech enhancement device is provided, which may include: an acquisition module, configured to acquire input speech; a processing module, configured to input the input speech into a speech enhancement model to obtain enhanced speech corresponding to the input speech, wherein the speech enhancement model is obtained based on the training method as described above.

[0023] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, which may include: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute the training method and speech enhancement method as described above.

[0024] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium storing instructions, which, when executed by at least one processor, prompts the at least one processor to execute the training method and speech enhancement method as described above.

[0025] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the training method and speech enhancement method as described above.

[0026] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0027] By introducing a neural network for determining speech speed and pitch changes during model training, the trained speech enhancement model can extract features related to speech speed and pitch changes, and then perform targeted speech enhancement processing based on the features, thereby improving the noise reduction effect of the speech enhancement model in speed and pitch changes scenarios, and enhancing the robustness of the speech enhancement model for speed and pitch changes.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0030] Figure 1 is a schematic diagram of a speech enhancement system according to an embodiment of the present disclosure;

[0031] Figure 2 is a flow chart of a speech enhancement method according to an embodiment of the present disclosure;

[0032] Figure 3 is a flow chart of a method for training a speech enhancement model according to an embodiment of the present disclosure;

[0033] Figure 4 is a structural schematic diagram of a speech enhancement system according to an embodiment of the present disclosure;

[0034] Figure 5 is a structural schematic diagram of a speech enhancement device according to an embodiment of the present disclosure;

[0035] Figure 6 is a block diagram of a speech enhancement device according to an embodiment of the present disclosure;

[0036] Figure 7 is a block diagram of a training device for a speech enhancement model according to an embodiment of the present disclosure;

[0037] Figure 8 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0038] Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures. DETAILED DESCRIPTION

[0039] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0040] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to assist in understanding, but these details are considered to be exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures are omitted for clarity and brevity.

[0041] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be clear to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the claims and their equivalents.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0043] In order to improve the performance of deep neural network noise reduction in speed and pitch change scenarios, this paper proposes a speed and pitch change-aware neural network speech noise reduction solution. The prediction of speed and pitch change values ​​is added to the existing noise reduction model training. Through multi-task training, the neural network can actively extract the speed and pitch change information of the input signal, thereby guiding the speech enhancement model of this paper to perform targeted noise reduction processing.

[0044] Hereinafter, according to various embodiments of the present disclosure, the method and apparatus of the present disclosure will be described in detail with reference to the accompanying drawings.

[0045] Figure 1 is a schematic diagram of a speech enhancement system according to an embodiment of the present disclosure. Figure 1A speed-changing and pitch-changing perceptible speech enhancement system framework according to an embodiment of the present disclosure is shown. A common speech enhancement network can be divided into two parts: a feature extraction network and a noise reduction network. According to an embodiment of the present disclosure, a speed-changing and pitch-changing determination network can be added after the feature extraction network. The speed-changing and pitch-changing determination network can estimate the speed change value and the pitch change value of the current speech based on the features output by the feature extraction network. By optimizing the accuracy of speed-changing and pitch-changing judgment, the features output by the feature extraction network can contain the speed-changing and pitch-changing information of the speech, and the noise reduction network can generate a targeted enhanced speech signal based on these speed-changing and pitch-changing information.

[0046] The speech enhancement model according to the present disclosure may include Figure 1 The feature extraction network and noise reduction network of the speech enhancement model are used. In the process of training the speech enhancement model, the noisy speech signal can be input to the feature extraction network to obtain the feature vector, and then the masking matrix and the speed change and pitch estimation value of the enhanced speech are obtained by passing through the noise reduction network and the speed change and pitch determination network respectively. The loss function value is calculated based on the output of the noise reduction network and the output of the speed change and pitch determination network. With the goal of minimizing the loss function value, the feature extraction network, the noise reduction network and the speed change and pitch determination network are optimized, and the model parameters are updated until the speech enhancement model converges.

[0047] The speech enhancement model trained according to the above method can perceive the speed change and pitch change information of speech.

[0048] In the model prediction stage, for the speech noise reduction scenario, you can only run Figure 1 The feature extraction network and noise reduction network are used to obtain the noise-reduced speech data. During the model testing process, the speed change and pitch change judgment network can be omitted. Figures 2 to 4 The model training process and model prediction process of the present invention are described in detail.

[0049] Figure 2 1 is a flow chart of a speech enhancement method according to an embodiment of the present disclosure. The speech enhancement method according to the present disclosure can be applied to any speech noise reduction scenario. For example, speech noise reduction can be performed on noisy speech without changing speed or pitch, noisy speech with changing speed or pitch, noisy speech with changing speed or pitch, and noisy speech with changing speed or pitch.

[0050] The speech enhancement method according to the present disclosure can be performed by any electronic device. The electronic device can be a terminal where a user is located. The electronic device can be at least one of a smart phone, a tablet computer, a portable computer, and a desktop computer. The electronic device can be installed with a target application for performing noise reduction processing on the above-mentioned noisy speech to obtain clean speech.

[0051] In step S201, an input speech is acquired. Here, the input speech may be a noisy speech without changing speed or pitch, a noisy speech with changing speed or pitch, a noisy speech with changing speed or pitch, or a noisy speech with changing speed or pitch, etc. The noise signal in the speech may include various noises.

[0052] In step S202, the acquired input speech is input into a speech enhancement model to obtain enhanced speech corresponding to the input speech.

[0053] The speech enhancement model according to the present disclosure may be implemented by a neural network, for example, the speech enhancement model may include a feature extraction network and a noise reduction network. The feature extraction network may be implemented by a multi-layer convolutional network and a recurrent neural network, and the noise reduction network may be implemented by a multi-layer recurrent neural network and a fully connected network. However, the above examples are merely exemplary, and the present disclosure is not limited thereto.

[0054] As an example, first, the input speech is subjected to a short-time Fourier transform (STFT) to obtain the input speech spectrum. The input speech spectrum is subjected to a multi-layer convolutional network and a recurrent neural network to extract the corresponding feature vector. The extracted feature vector is subjected to a multi-layer recurrent neural network and a fully connected network to output the corresponding time-frequency masking matrix. The time-frequency masking matrix is ​​multiplied by the input speech spectrum to obtain an estimated clean speech spectrum, and then the estimated clean speech spectrum is subjected to an inverse short-time Fourier transform (ISTFT) to obtain an estimated clean speech signal.

[0055] The following will refer to Figure 3 and Figure 4 How to train the speech enhancement model of the present disclosure is described in detail.

[0056] Figure 3 It is a flowchart of a method for training a speech enhancement model according to an embodiment of the present disclosure. Figure 4 It is a structural diagram of a speech enhancement system according to an embodiment of the present disclosure.

[0057] The training method according to the present disclosure can be performed by any electronic device. The electronic device can be a terminal where a user is located. The electronic device can be at least one of a smart phone, a tablet computer, a portable computer, and a desktop computer.

[0058] The speech enhancement model according to the embodiment of the present disclosure may include a feature extraction part and a noise reduction part. When training the speech enhancement model, a speed change and pitch change determination part may be introduced so that the trained speech enhancement model can perceive the speed change and pitch change information of the speech, so as to perform noise reduction processing on the input speech in a more targeted manner.

[0059] Reference Figure 3 In step S301, a training sample is obtained. The training sample may include a clean speech signal and a noisy speech signal. The clean speech signal may refer to a noise-free speech signal that has not been subjected to speed change and pitch change processing (i.e., a speech signal that is not subjected to speed change and pitch change processing), and the noisy speech signal may refer to a speech signal obtained by performing speed change and pitch change processing and noise mixing processing on the clean speech signal.

[0060] As an example, a clean speech signal can be collected in a quiet environment (such as a recording studio). Then, the clean speech signal is mixed with various noise signals to obtain a noisy speech signal, and the noisy speech signal is adjusted to a noisy speech signal with variable speed and pitch according to different speed changes, speed changes, speed increases and speed decreases. The maximum speed change, the maximum speed change, the maximum speed increase and the maximum speed decrease can be preset according to actual needs.

[0061] In addition, since speech rate and intonation information is long-term information, the sample data used in training can be speech signals of a longer time range, such as 4-5 seconds of speech signals, so that the features extracted later can be more accurate.

[0062] In step S302, information of the noisy speech signal is input into a feature extraction network of a speech enhancement model to obtain features corresponding to the noisy speech signal.

[0063] In the present disclosure, the frequency domain information of the speech signal can be used to train the speech enhancement model, and the time domain information of the speech signal can also be used to train the speech enhancement model.

[0064] When the information of the noisy speech signal is frequency domain information, STFT may be first performed on the noisy speech signal to obtain spectrum information of the noisy speech signal, and then the spectrum information is input into a feature extraction network to obtain feature information of the noisy speech signal.

[0065] Since the information of speech rate and intonation is long-term information, the feature extraction network needs to have a larger time domain perception range, so the feature extraction network can add structures such as recurrent neural networks for modeling.

[0066] As an example, see Figure 4 , the feature extraction network can be composed of a multi-layer convolutional network CNNs and a recurrent neural network GRUs_0. After obtaining the spectrum information of the noisy speech signal, the spectrum information can be input into the feature extraction network to obtain the features of the noisy speech signal.

[0067] In step S303, the obtained features are input into the denoising network of the speech enhancement model to obtain prediction information predicted by the denoising network. Here, the prediction information can be used to generate an enhanced speech signal of the noisy speech signal. The enhanced speech signal can refer to a clean speech estimation signal output by the speech enhancement model.

[0068] As an example, see Figure 4 , the denoising network can be implemented by using a multi-layer recurrent neural network GRUs_1 and a fully connected network MLPs_1. After obtaining the features of the noisy speech signal, the extracted features can be input into the denoising network, thereby obtaining information for obtaining an enhanced speech signal for the noisy speech signal.

[0069] The prediction information may be a masking matrix for obtaining an enhanced speech signal. Masking matrices in different domains may be obtained according to whether the information of the noisy speech signal is frequency domain information or time domain information. The denoising network may output a masking matrix for obtaining an enhanced speech signal based on the extracted features.

[0070] When the prediction information is a frequency domain masking matrix, the spectrum of the enhanced speech is obtained by multiplying the obtained masking matrix with the spectrum of the noisy speech.

[0071] In step S304, the obtained features are input into a speed change and pitch change determination network to obtain a speed change and pitch change estimation value of the noisy speech signal predicted by the speed change and pitch change determination network.

[0072] As an example, see Figure 4 , the speed change and pitch change judgment network can be implemented by a multi-layer recurrent neural network GRUs_2 and a fully connected network MLPs_2. After obtaining the features of the noisy speech signal, the extracted features can be input into the speed change and pitch change judgment network to obtain the speed change and pitch change estimation value for the noisy speech signal.

[0073] In step S305, the parameters of the feature extraction network, the noise reduction network and the speed and pitch determination network are adjusted based on the information of the noisy speech signal, the prediction information, the speed and pitch estimation value and the information of the clean speech signal to train the speech enhancement model.

[0074] Specifically, the first loss function can be calculated based on the information of the noisy speech signal, the prediction information and the information of the clean speech signal, the second loss function can be calculated based on the speed change and pitch change estimation value and the actual speed change and pitch change value when the speed change and pitch change processing is performed on the clean speech signal, and the speech enhancement model is trained by adjusting the parameters of the feature extraction network, the noise reduction network and the speed change and pitch change determination network according to the first loss function and the second loss function. The construction of the loss function is described below using frequency domain information as an example.

[0075] The masking matrix used to obtain the enhanced speech signal may be multiplied with the amplitude spectrum of the noisy speech signal to obtain the amplitude spectrum of the enhanced speech signal, and the first loss function may be calculated based on the amplitude spectrum of the enhanced speech signal and the amplitude spectrum of the clean speech signal.

[0076] As an example, the first loss function may be the mean square error between the enhanced speech signal amplitude spectrum and the clean speech signal amplitude spectrum. For example, the first loss function L is represented by the following equation (1): spec :

[0077]

[0078] Where t represents the tth frame of the speech signal, T represents the total number of frames of the speech signal, M(t) represents the masking vector of the tth frame, Y(t) represents the amplitude spectrum of the tth frame of the noisy speech signal, and X(t) represents the amplitude spectrum of the tth frame of the clean speech signal.

[0079] In the present disclosure, the speed change value and the pitch change value can be considered as two separate variables. A first error function can be calculated based on the speed change estimated value in the speed change pitch estimated value and the speed change actual value in the speed change pitch actual value, a second error function can be calculated based on the pitch change estimated value in the speed change pitch estimated value and the pitch change actual value in the speed change pitch actual value, and then a second loss function for speed change pitch determination can be calculated based on the first error function and the second error function.

[0080] As an example, the second loss function can be expressed by the following equation (2):

[0081]

[0082] Where t represents the tth frame of the speech signal, T represents the total number of frames of the speech signal, and S norm (t) and P norm (t) respectively represent the actual speed change value and actual pitch change value corresponding to the t-th frame, and They respectively represent the speed change estimation value and pitch change estimation value corresponding to the t-th frame, and the function Dist(x, y) represents the error function of the speed change value or the pitch change value.

[0083] Here, Dist(x, y) = max(|xy|-δ, 0). δ represents the allowable error threshold of the speed change value or the pitch change value. When the estimated error is less than δ, the error function value is 0. The purpose of designing the allowable error threshold is that the speech itself has a certain range of speed and intonation changes, and the neural network's estimation error calculation of the speed change and pitch change value needs to take this part of the change into account. This ensures that the trained model is more in line with actual requirements.

[0084] After obtaining the first loss function and the second loss function, a weighted summation may be performed on the first loss function and the second loss function, and the speech enhancement model may be trained by minimizing the final loss function value.

[0085] According to another embodiment of the present disclosure, in order to make the speed change and modulation value estimated by the speed change and modulation determination network more accurate, the speed change and modulation value can be regularized during training. Specifically, the speed change and modulation estimation value and the corresponding speed change and modulation actual value can be adjusted to values ​​within a preset range, and then the second loss function is calculated based on the error between the adjusted speed change and modulation estimation value and the adjusted speed change and modulation actual value.

[0086] As an example, based on the maximum variable speed multiple, the maximum variable speed multiple and the preset variable speed value adjustment strategy, the variable speed estimated value in the variable speed modulation estimated value and the variable speed actual value in the variable speed modulation actual value can be adjusted to a value within a preset range. Based on the maximum up-modulation multiple, the maximum down-modulation multiple and the preset modulation adjustment strategy, the modulation estimated value in the variable speed modulation estimated value and the modulation actual value in the variable speed modulation actual value can be adjusted to a value within a preset range. The maximum variable speed multiple, the maximum variable speed multiple, the maximum up-modulation multiple and the maximum down-modulation multiple can be set differently according to actual needs.

[0087] The preset range of the speed change value can be -1 to the ratio of the maximum faster speed to the maximum slower speed, and the preset range of the pitch change value can be -1 to the ratio of the maximum up-pitch speed to the maximum down-pitch speed.

[0088] Assuming that the original speed change value of the speech signal is S, and the maximum speed change times and the maximum speed change times are both N, then the range of the original speed change value S is In order to make the speed change value more linearly reflect the audio characteristics, the regularized speed change value S norm It can be expressed as the following equation (3):

[0089]

[0090] According to the speed adjustment strategy shown in equation (3), S norm The range is [-1, 1].

[0091] When the maximum speed change is N and the maximum speed change is M, the adjusted speed change value S norm It can be expressed as the following equation (4):

[0092]

[0093] According to the speed value adjustment strategy shown in equation (4), S norm The range is

[0094] Similarly, assuming that the original pitch change value of the speech signal is P, and the maximum pitch increase rate and the maximum pitch decrease rate are both N, then the regularized pitch change value P norm It can be expressed as the following equation (5)

[0095]

[0096] According to the pitch adjustment strategy shown in equation (5), P norm The range is [-1, 1].

[0097] When the maximum up-pitch speed is N and the maximum down-pitch speed is M, the adjusted pitch change value P norm It can be expressed as the following equation (6):

[0098]

[0099] According to the pitch adjustment strategy shown in equation (6), P norm The range is

[0100] When the second loss function is calculated using the regularized speed / modulation value, for the allowable error threshold δ in the error function, assuming that the speed / modulation range is When the speech speed / intonation variation within 20% is allowed to be excluded from the error, the allowable error threshold of the regularized speed / intonation value can be expressed as

[0101] Next, according to the above equation (2), the second loss function is calculated using the regularized speed change and modulation values.

[0102] According to another example of the present disclosure, in the calculation of the second loss function, considering that non-speech frames do not have the concept of speed change and pitch change, the influence of non-speech frames can be removed in the calculation of the second loss function. The second loss function can be calculated using the speed change and pitch change estimation value and the speed change and pitch change actual value for the speech frame. The prediction accuracy of the speed change and pitch change judgment network trained in this way is higher.

[0103] For example, the second loss function considering the influence of non-speech frames can be expressed as follows:

[0104]

[0105] Among them, V(t) represents the judgment function of whether the tth frame is speech,

[0106]

[0107] After obtaining the first loss function and the second loss function, the first loss function and the second loss function may be combined to obtain a total loss function.

[0108] As an example, the loss functions of equation (1) and equation (7) are used to construct the final loss function L all , as shown in equation (8):

[0109]

[0110] Wherein, a represents the weight of the second loss function of the speed change and pitch change judgment network, for example, a=0.1.

[0111] In model training, the goal is to minimize the total loss function value, optimize the parameters of the feature extraction network, denoising network and speed and pitch change determination network, and update the model parameters until the model converges.

[0112] also, Figure 4 The structure of single-channel (one speech enhancement model) frequency domain noise reduction is shown, however, according to the embodiment of the present disclosure, it can be Figure 3 The training method can simultaneously train different speech enhancement models for different types of noise reduction tasks. The training method disclosed herein can be applicable to single-channel time domain and frequency domain noise reduction systems, and can also be applicable to multi-channel (multiple speech enhancement models for different noise reduction tasks) time domain and frequency domain noise reduction systems. When multiple speech enhancement models are trained simultaneously, a speed change and pitch change determination network can be shared, or a speed change and pitch change determination network can be set for each speech enhancement model.

[0113] According to the characteristics of short video scenes, this paper designs a neural network noise reduction system that can perceive speed and pitch changes. Through multi-task training, the neural network can extract feature vectors related to speed and pitch changes, and then perform targeted speech noise reduction processing based on the feature vectors, thereby improving the noise reduction effect of the neural network in speed and pitch change scenes.

[0114] Figure 5 It is a structural diagram of a speech enhancement device in a hardware operating environment of an embodiment of the present disclosure.

[0115] like Figure 5 As shown, the speech enhancement device 500 may include: a processing component 501, a communication bus 502, a network interface 503, an input / output interface 504, a memory 505, and a power supply component 506. The communication bus 502 is used to realize the connection and communication between these components. The input / output interface 504 may include a video display (such as a liquid crystal display), a microphone and a speaker, and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.). Optionally, the input / output interface 504 may also include a standard wired interface and a wireless interface. The network interface 503 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity interface). The memory 505 may be a high-speed random access memory or a stable non-volatile memory. The memory 505 may also be a storage device independent of the aforementioned processing component 501.

[0116] Those skilled in the art will understand that Figure 5The structure shown in the figure does not constitute a limitation on the speech enhancement device 500, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0117] like Figure 5 As shown, the memory 505 as a storage medium may include an operating system (such as a MAC operating system), a data storage module, a network communication module, a user interface module, a model training program, a speech enhancement program, and a database.

[0118] exist Figure 5 In the voice enhancement device 500 shown, the network interface 503 is mainly used for data communication with external electronic devices / terminals; the input and output interface 504 is mainly used for data interaction with users; the processing component 501 and the memory 505 in the voice enhancement device 500 can be set in the voice enhancement device 500, and the voice enhancement device 500 calls the model training program, the voice enhancement program and various APIs provided by the operating system stored in the memory 505 through the processing component 501 to execute the model training method and the voice enhancement method provided in the embodiments of the present disclosure.

[0119] The processing component 501 may include at least one processor, and a computer executable instruction set is stored in the memory 505. When the computer executable instruction set is executed by at least one processor, the model training method and the speech enhancement method according to the embodiment of the present disclosure are executed. However, the above examples are only exemplary, and the present disclosure is not limited thereto.

[0120] The processing component 501 can control the components included in the speech enhancement device 500 by executing a program.

[0121] The speech enhancement device 500 may receive or output a speech signal via the input / output interface 504 .

[0122] The processing component 501 can execute the model training method and the speech enhancement method according to the embodiments of the present disclosure.

[0123] As an example, the speech enhancement device 500 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the speech enhancement device 500 is not necessarily a single electronic device, but may also be any collection of devices or circuits capable of executing the above instructions (or instruction sets) individually or in combination. The speech enhancement device 500 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0124] In the speech enhancement device 500, the processing component 501 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processing component 501 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0125] The processing component 501 can run instructions or codes stored in the memory, wherein the memory 505 can also store data. Instructions and data can also be sent and received over the network via the network interface 503, wherein the network interface 503 can use any known transmission protocol.

[0126] The memory 505 may be integrated with the processing component 501, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 505 may include a separate device, such as an external disk drive, a storage array, or any other storage device that may be used by a database system. The memory and the processing component 501 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processing component 501 can read data stored in the memory 505.

[0127] Figure 6 is a block diagram of a speech enhancement apparatus according to an embodiment of the present disclosure.

[0128] Reference Figure 6 , the speech enhancement device 600 may include an acquisition module 601 and a processing module 602. Each module in the speech enhancement device 600 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of the module. In various embodiments, some modules in the speech enhancement device 600 may be omitted, or additional modules may be further included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus may equivalently perform the functions of the corresponding modules / elements before the combination.

[0129] The acquisition module 601 can acquire input speech.

[0130] The processing module 602 may input the input speech into a speech enhancement model to obtain enhanced speech corresponding to the input speech, wherein the speech enhancement model is obtained based on the training method as described above.

[0131] The above is based on Figures 2 to 4 The speech enhancement operation and model training operation are described in detail and will not be described again here.

[0132] Figure 7 is a block diagram of a training device for a speech enhancement model according to an embodiment of the present disclosure.

[0133] Reference Figure 7 , the training device 700 may include an acquisition module 701, a feature extraction module 702, a noise reduction module 703, a speed change and pitch change determination module 704, and a training module 705. Each module in the training device 700 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of module. In various embodiments, some modules in the training device 700 may be omitted, or additional modules may be included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus the functions of the corresponding modules / elements before the combination may be equivalently performed.

[0134] The acquisition module 701 may acquire training samples. The training samples may include clean speech signals and noisy speech signals. The clean speech signals are noise-free speech signals that have not been subjected to speed and pitch change processing, and the noisy speech signals are obtained by performing speed and pitch change processing and noise mixing processing on the clean speech signals.

[0135] The feature extraction module 702 may input information of the noisy speech signal into a feature extraction network to obtain features corresponding to the noisy speech signal.

[0136] The noise reduction module 703 may input the features into the noise reduction network to obtain prediction information predicted by the noise reduction network. The prediction information may be used to generate an enhanced speech signal of the noisy speech signal.

[0137] The speed change and pitch change determination module 704 may input the features into the speed change and pitch change determination network to obtain the speed change and pitch change estimation value of the noisy speech signal predicted by the speed change and pitch change determination network.

[0138] The training module 705 can adjust the parameters of the feature extraction network, the noise reduction network and the speed and pitch determination network based on the information of the noisy speech signal, the prediction information, the speed and pitch estimation value and the clean speech signal information to train the speech enhancement model.

[0139] Optionally, the training module 705 can calculate a first loss function based on the information of the noisy speech signal, the prediction information and the information of the clean speech signal; calculate a second loss function based on the speed change and pitch change estimation value and the actual value of the speed change and pitch change when the speed change and pitch change processing is performed on the clean speech signal; and adjust the parameters of the feature extraction network, the denoising network and the speed change and pitch change determination network according to the first loss function and the second loss function.

[0140] Optionally, when the information of the noisy speech signal may include the amplitude spectrum of the noisy speech signal, and the prediction information may include a masking matrix for obtaining an enhanced speech signal, the training module 705 may multiply the masking matrix for obtaining the enhanced speech signal and the amplitude spectrum of the noisy speech signal to obtain the amplitude spectrum of the enhanced speech signal, and calculate the first loss function based on the amplitude spectrum of the enhanced speech signal and the amplitude spectrum of the clean speech signal.

[0141] Optionally, the training module 705 may calculate a first error function based on the speed shift estimated value in the speed shift pitch estimated value and the speed shift actual value in the speed shift pitch actual value; calculate a second error function based on the pitch estimated value in the speed shift pitch estimated value and the pitch actual value in the speed shift pitch actual value; and calculate a second loss function based on the first error function and the second error function.

[0142] Optionally, the training module 705 may adjust the speed change modulation estimate value and the speed change modulation actual value to values ​​within a preset range; and calculate the second loss function based on the error between the adjusted speed change modulation estimate value and the adjusted speed change modulation actual value.

[0143] Optionally, the training module 705 can obtain the maximum faster speed, maximum slower speed, maximum up-modulation speed and maximum down-modulation speed when performing speed and pitch changing processing on a clean speech signal; based on the maximum faster speed, maximum slow speed and a preset speed value adjustment strategy, the speed estimated value in the speed and pitch changing estimated value and the speed actual value in the speed and pitch changing actual value are adjusted to values ​​within a preset range; based on the maximum up-modulation speed, maximum down-modulation speed and a preset pitch adjustment strategy, the pitch estimated value in the speed and pitch changing estimated value and the pitch actual value in the speed and pitch changing actual value are adjusted to values ​​within a preset range.

[0144] Optionally, the training module 705 may calculate the second loss function using the speed change and pitch change estimation value and the speed change and pitch change actual value for the speech frame.

[0145] Optionally, the information of the noisy speech signal may include frequency domain information of the noisy speech signal, and the prediction information may include frequency domain information for obtaining an enhanced speech signal; or, the information of the noisy speech signal may include time domain information of the noisy speech signal, and the prediction information may include time domain information for obtaining an enhanced speech signal. That is, the speech enhancement model of the present disclosure may be applicable to enhancing speech time domain signals, and may also be applicable to enhancing speech frequency domain signals.

[0146] The above is based on Figures 2 to 4 The model training operation has been described in detail and will not be described here.

[0147] According to an embodiment of the present disclosure, an electronic device may be provided. Figure 8It is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 800 may include at least one memory 802 and at least one processor 801. The at least one memory 802 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 801, the model training method and the speech enhancement method according to the embodiment of the present disclosure are executed.

[0148] The processor 801 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 801 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0149] The memory 802 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, a model training program, a speech enhancement program, and a database.

[0150] The memory 802 may be integrated with the processor 801, for example, RAM or flash memory may be arranged within an integrated circuit microprocessor or the like. In addition, the memory 802 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 802 and the processor 801 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 801 can read files stored in the memory 802.

[0151] In addition, the electronic device 800 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 800 may be connected to each other via a bus and / or a network.

[0152] Those skilled in the art will understand that Figure 8 The structures shown in the drawings do not constitute limitations of the drawings and may include more or less components than shown in the drawings, or combine certain components, or arrange the components differently.

[0153] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein, when the instructions are executed by at least one processor, the at least one processor is prompted to execute the model training method and speech enhancement method according to the present disclosure. Examples of computer-readable storage media include: read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as multimedia card, secure digital card, etc.), etc. Digital, SD) card or ultra-fast digital card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-temporary manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in computer devices such as clients, hosts, proxy devices, servers, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0154] According to an embodiment of the present disclosure, a computer program product may also be provided, and the instructions in the computer program product may be executed by a processor of a computer device to complete the above-mentioned model training method and speech enhancement method.

[0155] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0156] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a speech enhancement model, characterized in that: The speech enhancement model includes a feature extraction network and a noise reduction network, wherein the training method includes: Acquire a training sample, wherein the training sample includes a clean speech signal, a noisy speech signal, and an actual value of speed change and modulation when speed change and modulation modulation processing is performed on the clean speech signal, wherein the clean speech signal is a noise-free speech signal that has not been processed by speed change and modulation, and the noisy speech signal is obtained by performing speed change and modulation processing and noise mixing processing on the clean speech signal; Inputting information of the noisy speech signal into the feature extraction network to obtain features corresponding to the noisy speech signal; Inputting the feature into the noise reduction network to obtain prediction information predicted by the noise reduction network, wherein the prediction information is used to generate an enhanced speech signal of the noisy speech signal; Inputting the feature into a speed change and pitch change determination network to obtain a speed change and pitch change estimation value of the noisy speech signal predicted by the speed change and pitch change determination network; Adjusting the speed change modulation estimation value and the speed change modulation actual value to values ​​within a preset range; and Based on the information of the noisy speech signal, the prediction information, the adjusted speed change and pitch change estimation value, the adjusted speed change and pitch change actual value and the information of the clean speech signal, the parameters of the feature extraction network, the noise reduction network and the speed change and pitch change determination network are adjusted to train the speech enhancement model.

2. The training method according to claim 1, characterized in that: The method further comprises adjusting parameters of the feature extraction network, the noise reduction network and the speed and pitch change determination network based on the information of the noisy speech signal, the prediction information, the adjusted speed and pitch change estimation value, the adjusted speed and pitch change actual value and the information of the clean speech signal to train the speech enhancement model, including: Calculating a first loss function based on the information of the noisy speech signal, the prediction information, and the information of the clean speech signal; Calculating a second loss function according to the adjusted speed change and modulation estimation value and the adjusted speed change and modulation actual value; The parameters of the feature extraction network, the noise reduction network and the speed and pitch change determination network are adjusted according to the first loss function and the second loss function.

3. The training method according to claim 2, characterized in that: The information of the noisy speech signal includes an amplitude spectrum of the noisy speech signal, and the prediction information includes a masking matrix for obtaining an enhanced speech signal. The calculating a first loss function according to the information of the noisy speech signal, the prediction information and the information of the clean speech signal comprises: Multiplying the masking matrix for obtaining the enhanced speech signal and the amplitude spectrum of the noisy speech signal to obtain the amplitude spectrum of the enhanced speech signal; The first loss function is calculated according to the amplitude spectrum of the enhanced speech signal and the amplitude spectrum of the clean speech signal.

4. The training method according to claim 2, characterized in that: Calculating a second loss function according to the adjusted speed change and modulation estimation value and the adjusted speed change and modulation actual value includes: calculating a first error function based on the speed change estimated value in the adjusted speed change pitch estimated value and the speed change actual value in the adjusted speed change pitch actual value; calculating a second error function based on the estimated pitch value in the adjusted speed change pitch estimated value and the actual pitch value in the adjusted speed change pitch actual value; The second loss function is calculated based on the first error function and the second error function.

5. The training method according to claim 1, characterized in that: The speed change modulation estimation value and the speed change modulation actual value are adjusted to values ​​within a preset range, including: Acquire a maximum speed increase, a maximum speed decrease, a maximum pitch increase, and a maximum pitch decrease when performing speed and pitch change processing on the clean speech signal; Based on the maximum faster speed, the maximum slower speed and a preset speed value adjustment strategy, the speed estimated value in the speed modulation estimated value and the speed actual value in the speed modulation actual value are adjusted to speed values ​​within a preset range; Based on the maximum up-modulation speed, the maximum down-modulation speed and a preset pitch adjustment strategy, the pitch estimation value in the speed-changing pitch estimation value and the pitch actual value in the speed-changing pitch actual value are adjusted to pitch values ​​within a preset range.

6. The training method according to claim 2, characterized in that: Calculating a second loss function according to the adjusted speed change and modulation estimation value and the adjusted speed change and modulation actual value includes: The second loss function is calculated using the speed-shifted pitch estimation value and the speed-shifted pitch actual value for the speech frame.

7. The training method according to claim 1, characterized in that: The information of the noisy speech signal includes frequency domain information of the noisy speech signal, and the prediction information includes frequency domain information for obtaining an enhanced speech signal; Alternatively, the information of the noisy speech signal includes time domain information of the noisy speech signal, and the prediction information includes time domain information used to obtain an enhanced speech signal.

8. A speech enhancement method, characterized in that: include: Get input voice; Inputting the input speech into a speech enhancement model to obtain enhanced speech corresponding to the input speech, Wherein, the speech enhancement model is obtained based on the training method described in any one of claims 1 to 7.

9. A training device for a speech enhancement model, characterized in that: The speech enhancement model includes a feature extraction network and a noise reduction network, wherein the training device includes: an acquisition module, configured to acquire training samples, wherein the training samples include a clean speech signal, a noisy speech signal, and an actual value of a speed change and modulation when a speed change and modulation processing is performed on the clean speech signal, wherein the clean speech signal is a noise-free speech signal that has not been subjected to a speed change and modulation processing, and the noisy speech signal is obtained by performing a speed change and modulation processing and a noise mixing processing on the clean speech signal; A feature extraction module, configured to input information of the noisy speech signal into the feature extraction network to obtain features corresponding to the noisy speech signal; a noise reduction module, configured to input the feature into the noise reduction network to obtain prediction information predicted by the noise reduction network, wherein the prediction information is used to generate an enhanced speech signal of the noisy speech signal; A speed change and pitch change determination module is configured to input the feature into a speed change and pitch change determination network to obtain a speed change and pitch change estimation value of the noisy speech signal predicted by the speed change and pitch change determination network; The training module is configured to: adjust the speed change and pitch change estimation value and the speed change and pitch change actual value to values ​​within a preset range; adjust the parameters of the feature extraction network, the noise reduction network and the speed change and pitch change determination network based on the information of the noisy speech signal, the prediction information, the adjusted speed change and pitch change estimation value, the adjusted speed change and pitch change actual value and the information of the clean speech signal to train the speech enhancement model.

10. The training device according to claim 9, characterized in that The training module is configured as: Calculating a first loss function based on the information of the noisy speech signal, the prediction information, and the information of the clean speech signal; Calculating a second loss function according to the adjusted speed change and modulation estimation value and the adjusted speed change and modulation actual value; The parameters of the feature extraction network, the noise reduction network and the speed and pitch change determination network are adjusted according to the first loss function and the second loss function.

11. The training device according to claim 10, characterized in that The information of the noisy speech signal includes an amplitude spectrum of the noisy speech signal, and the prediction information includes a masking matrix for obtaining an enhanced speech signal. Among them, the training module is configured as: Multiplying the masking matrix for obtaining the enhanced speech signal and the amplitude spectrum of the noisy speech signal to obtain the amplitude spectrum of the enhanced speech signal; The first loss function is calculated according to the amplitude spectrum of the enhanced speech signal and the amplitude spectrum of the clean speech signal.

12. The training device according to claim 10, characterized in that The training module is configured as: calculating a first error function based on the speed change estimated value in the adjusted speed change pitch estimated value and the speed change actual value in the adjusted speed change pitch actual value; calculating a second error function based on the estimated pitch value in the adjusted speed change pitch estimated value and the actual pitch value in the adjusted speed change pitch actual value; The second loss function is calculated based on the first error function and the second error function.

13. The training device according to claim 9, characterized in that The training module is configured as: Acquire a maximum speed increase, a maximum speed decrease, a maximum pitch increase, and a maximum pitch decrease when performing speed and pitch change processing on the clean speech signal; Based on the maximum faster speed, the maximum slower speed and a preset speed value adjustment strategy, the speed estimated value in the speed modulation estimated value and the speed actual value in the speed modulation actual value are adjusted to speed values ​​within a preset range; Based on the maximum up-modulation speed, the maximum down-modulation speed and a preset pitch value adjustment strategy, the pitch estimation value in the speed-changing pitch estimation value and the pitch actual value in the speed-changing pitch actual value are adjusted to pitch values ​​within a preset range.

14. The training device according to claim 10, characterized in that The training module is configured as: The second loss function is calculated using the speed-shifted pitch estimation value and the speed-shifted pitch actual value for the speech frame.

15. The training device according to claim 9, characterized in that The information of the noisy speech signal includes frequency domain information of the noisy speech signal, and the prediction information includes frequency domain information for obtaining an enhanced speech signal; Alternatively, the information of the noisy speech signal includes time domain information of the noisy speech signal, and the prediction information includes time domain information used to obtain an enhanced speech signal.

16. A speech enhancement device, characterized in that: include: An acquisition module, configured to acquire input speech; a processing module configured to input the input speech into a speech enhancement model to obtain enhanced speech corresponding to the input speech, Wherein, the speech enhancement model is obtained based on the training method described in any one of claims 1 to 7.

17. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 8.

18. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 8.

19. A computer program product, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the method according to any one of claims 1 to 8.

Images