A method, device, equipment and storage medium for speech enhancement

Abstract

The present application proposes a speech enhancement method, device, equipment and storage medium, the method comprising: inputting noisy speech into a pre-trained composite adversarial enhancement network model, obtaining a speech enhancement result output by the composite adversarial enhancement network model, the speech enhancement result at least including enhanced speech; wherein the voiceprint features of the enhanced speech obtained by the composite adversarial enhancement network model performing speech enhancement processing on the noisy speech are close to the voiceprint features of the clean speech corresponding to the noisy speech. The above method can reduce the speech distortion caused by speech enhancement while performing speech enhancement on the noisy speech.

Description

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on August 20, 2021, with application number 202110961480.8 and invention name “A speech enhancement method, device, equipment and storage medium”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of speech enhancement technology, and in particular to a speech enhancement method, apparatus, device and storage medium. Background Art

[0003] Speech enhancement refers to a technology that uses certain methods to filter out noise when clean speech is interfered with by various noises in real life, in order to improve the quality and intelligibility of speech.

[0004] For the speaker's voice, its main energy is concentrated in the low-frequency range (0Hz-1500Hz), and the high-frequency part (greater than 1500Hz) contains relatively less energy. Although the high-frequency part of the speaker's voice has low energy, it contains important voiceprint information such as the speaker's timbre and pitch. Compared with the high-energy low-frequency part, the low-energy high-frequency content is more easily submerged by noise.

[0005] Conventional speech enhancement solutions are usually signal processing-based speech enhancement methods and deep learning-based speech enhancement algorithms. These methods are more effective in enhancing speech signals with high signal-to-noise ratios. However, for speech signals with low signal-to-noise ratios, these methods will remove the high-frequency part of the speech signal together with the noise when removing noise, causing the speech signal to lose important voiceprint information and resulting in audible speech distortion. Summary of the invention

[0006] Based on the above technical status, the present application proposes a speech enhancement method, apparatus, device and storage medium, which can reduce the speech distortion caused by speech enhancement while enhancing noisy speech.

[0007] In order to achieve the above objectives, this application proposes a specific technical solution:

[0008] A speech enhancement method, comprising:

[0009] Inputting noisy speech into a pre-trained composite adversarial enhancement network model to obtain a speech enhancement result output by the composite adversarial enhancement network model, wherein the speech enhancement result at least includes enhanced speech;

[0010] The voiceprint features of the enhanced speech obtained by performing speech enhancement processing on the noisy speech by the composite adversarial enhancement network model are close to the voiceprint features of the clean speech corresponding to the noisy speech.

[0011] Optionally, the speech enhancement result also includes an audio gain, and the audio gain is determined based on an amplitude ratio of the enhanced speech obtained by performing speech enhancement processing on the noisy speech to the noisy speech at each sampling point;

[0012] The audio gain obtained by the composite adversarial enhancement network model performing speech enhancement processing on the noisy speech is close to the audio gain of the clean speech corresponding to the noisy speech relative to the noisy speech;

[0013] The method further comprises:

[0014] According to the speech enhancement result, an enhanced speech corresponding to the input noisy speech is obtained.

[0015] Optionally, acquiring, according to the speech enhancement result, an enhanced speech corresponding to the input noisy speech includes:

[0016] According to the enhanced speech and the audio gain in the speech enhancement result, the enhanced speech corresponding to the input noisy speech is calculated.

[0017] Optionally, the composite adversarial enhancement network model includes at least two deep adversarial enhancement modules, the deep adversarial enhancement modules are connected in series in sequence, each deep adversarial enhancement module is used to perform speech enhancement processing, and outputs enhanced speech obtained by the speech enhancement processing.

[0018] Optionally, the composite adversarial enhancement network includes a first deep adversarial enhancement module, a second deep adversarial enhancement module and a third deep adversarial enhancement module connected in series in sequence;

[0019] Among them, the noisy speech input into the composite adversarial enhancement network model is input into the first deep adversarial enhancement module, and the output of the third deep adversarial enhancement module is used as the speech enhancement result output by the composite adversarial enhancement network model; and the difference between the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module and the signal-to-noise ratio of the input speech is the same as the difference between the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module and the signal-to-noise ratio of the input speech.

[0020] Optionally, the first depth adversarial enhancement module, the second depth adversarial enhancement module and the third depth adversarial enhancement module respectively include a set number of multi-scale residual units.

[0021] Optionally, acquiring, according to the speech enhancement result, an enhanced speech corresponding to the input noisy speech includes:

[0022] Performing weighted summation on the enhanced speech output by each deep adversarial enhancement module to obtain a first enhancement result;

[0023] Calculating a second enhancement result according to the noisy speech and the audio gain;

[0024] According to the first enhancement result and the second enhancement result, an enhanced speech corresponding to the input noisy speech is calculated.

[0025] Optionally, the training process of the composite adversarial enhanced network model includes:

[0026] Acquire a noisy speech sample, a clean speech corresponding to the noisy speech sample, and a target audio gain determined based on the noisy speech sample and the corresponding clean speech;

[0027] Inputting the noisy speech sample into the composite adversarial enhancement network model to obtain enhanced speech and audio gain output by the composite adversarial enhancement network model;

[0028] Acquire the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model, and acquire the voiceprint features of the clean speech corresponding to the noisy speech sample;

[0029] Determine an audio mapping loss according to the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample, determine an audio gain loss according to the audio gain output by the composite adversarial enhancement network model and the target audio gain, and determine a voiceprint feature loss according to the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and the voiceprint features of the clean speech corresponding to the noisy speech sample;

[0030] Parameters of the composite adversarial enhancement network model are corrected at least according to the audio mapping loss, the audio gain loss and the voiceprint feature loss.

[0031] Optionally, the method further includes:

[0032] Determining the speech difference loss by comparing the enhanced speech output by the composite adversarial enhancement network model with the clean speech corresponding to the noisy speech sample;

[0033] The step of performing parameter correction on the composite adversarial enhancement network model at least according to the audio mapping loss, the audio gain loss, and the voiceprint feature loss includes:

[0034] According to the audio mapping loss, the audio gain loss, the voiceprint feature loss and the speech difference loss, parameters of the composite adversarial enhancement network model are corrected.

[0035] Optionally, the obtaining of the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model, and the obtaining of the voiceprint features of the clean speech corresponding to the noisy speech sample, include:

[0036] Inputting the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample into a pre-trained voiceprint feature extraction network respectively, to obtain the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and the voiceprint features of the clean speech corresponding to the noisy speech sample;

[0037] The voiceprint feature extraction network is obtained by extracting the speaker's voiceprint features from the input speech and performing speaker recognition training.

[0038] Optionally, determining the speech difference loss by comparing the enhanced speech output by the composite adversarial enhancement network model with the clean speech corresponding to the noisy speech sample includes:

[0039] Using a pre-trained speech discrimination network, the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample are discriminated;

[0040] Determining the speech difference loss according to the discrimination result output by the speech discrimination network;

[0041] The speech discrimination network has the ability to distinguish between noisy speech and clean speech corresponding to the noisy speech.

[0042] Optionally, the composite adversarial enhancement network includes a first deep adversarial enhancement module, a second deep adversarial enhancement module and a third deep adversarial enhancement module connected in series in sequence; wherein the noisy speech input into the composite adversarial enhancement network model is input into the first deep adversarial enhancement module, and the output of the third deep adversarial enhancement module is used as the speech enhancement result output by the composite adversarial enhancement network model;

[0043] The parameter correction of the composite adversarial enhancement network model includes:

[0044] The parameters of the composite adversarial enhancement network model are corrected with the goal of increasing the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module by a first preset value relative to the signal-to-noise ratio of its input speech, and increasing the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module by the first preset value relative to the signal-to-noise ratio of its input speech.

[0045] A speech enhancement device, comprising:

[0046] A speech enhancement unit, used for inputting noisy speech into a pre-trained composite adversarial enhancement network model to obtain a speech enhancement result output by the composite adversarial enhancement network model, wherein the speech enhancement result at least includes enhanced speech;

[0047] The voiceprint features of the enhanced speech obtained by performing speech enhancement processing on the noisy speech by the composite adversarial enhancement network model are close to the voiceprint features of the clean speech corresponding to the noisy speech.

[0048] A speech enhancement device, comprising:

[0049] Memory and processor;

[0050] The memory is connected to the processor and is used to store programs;

[0051] The processor is used to implement the above-mentioned speech enhancement method by running the program in the memory.

[0052] A storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned speech enhancement method is implemented.

[0053] The speech enhancement method proposed in the present application performs speech enhancement processing on noisy speech with the help of a pre-trained composite adversarial enhancement network model to obtain enhanced speech. The composite adversarial enhancement network model has the ability to perform speech enhancement processing on noisy speech to obtain enhanced speech corresponding to the noisy speech, and the voiceprint characteristics of the enhanced speech obtained by the model through speech enhancement processing on noisy speech are close to the voiceprint characteristics of the clean speech corresponding to the noisy speech. Then, by processing noisy speech with the composite adversarial enhancement network model, not only can the purpose of speech enhancement of noisy speech be achieved, but also more voiceprint information in the speech can be retained, thereby reducing the speech distortion caused by speech enhancement.

[0054] Especially when performing speech enhancement on noisy speech with a low signal-to-noise ratio, the technical solution of the embodiment of the present application can more fully retain the voiceprint information in the speech signal while performing speech enhancement on the noisy speech, thereby solving the problem of obvious speech distortion caused by the existing speech enhancement scheme when performing speech enhancement on noisy speech with a low signal-to-noise ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0056] Figure 1 It is a flowchart of a speech enhancement method provided in an embodiment of the present application;

[0057] Figure 2 It is a structural diagram of a composite adversarial enhancement network model provided in an embodiment of the present application;

[0058] Figure 3 is a schematic diagram of the structure of a deep adversarial enhancement module provided in an embodiment of the present application;

[0059] Figure 4 is a schematic diagram of the structure of a multi-scale residual unit provided in an embodiment of the present application;

[0060] Figure 5 It is a schematic diagram of the training process of the composite adversarial enhancement network model provided in an embodiment of the present application;

[0061] Figure 6 is a structural schematic diagram of a speech enhancement device provided in an embodiment of the present application;

[0062] Figure 7 It is a structural schematic diagram of a speech enhancement device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solution of the embodiment of the present application is applicable to the field of speech enhancement technology. By applying the technical solution of the embodiment of the present application, noisy speech can be enhanced to obtain clearer and cleaner enhanced speech, and the enhanced speech can retain the high-frequency part of the speech signal, that is, retain the voiceprint information such as timbre and pitch in the speech signal.

[0064] Existing single-channel speech enhancement algorithms are mainly divided into signal processing-based speech enhancement methods and deep learning-based speech enhancement algorithms. Signal processing-based speech enhancement algorithms mainly use signal processing methods such as filtering for noise reduction. Due to the existence of many unreasonable assumptions, such as the assumption that speech and noise are independent of each other and that noise conforms to a stationary distribution, their performance ceiling is limited and their ability to suppress non-stationary noise signals is weak. Deep learning-based speech enhancement methods learn the statistical characteristics of speech and noise through supervised training, and their noise reduction performance is improved compared to signal processing-based speech enhancement algorithms. However, in complex scenarios such as low signal-to-noise ratio, noise and speech are deeply coupled, and when the model removes noise, it usually causes the loss of part of the speech, resulting in speech distortion.

[0065] Specifically, for a speaker's voice, the main energy is concentrated in the low-frequency range (0Hz-1500Hz), and the high-frequency part (greater than 1500Hz) contains relatively less energy. Although the high-frequency part of a speaker's voice has low energy, it contains important voiceprint information such as the speaker's timbre and pitch. Compared with the high-energy low-frequency part, the low-energy high-frequency content is more easily drowned by noise.

[0066] In low signal-to-noise ratio scenarios, existing speech enhancement methods will remove the high-frequency part of the speech together with the noise, resulting in the loss of the high-frequency part of the speech and important voiceprint information such as the speaker's timbre and pitch, causing speech distortion.

[0067] Therefore, existing speech enhancement methods cannot fully preserve the voiceprint information in the speech, especially when enhancing noisy speech with low signal-to-noise ratio, it is easy to cause speech distortion.

[0068] In view of the above problems, an embodiment of the present application proposes a speech enhancement solution, which can enhance noisy speech to obtain enhanced speech while enabling the enhanced speech to retain more voiceprint information, thereby reducing speech distortion caused by speech enhancement and improving the speech enhancement effect.

[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] This application embodiment proposes a method for speech enhancement. Figure 1 As shown, the method includes:

[0071] S101: Acquire noisy speech.

[0072] Specifically, the noisy speech is a speech signal composed of a mixture of clean speech and noise. The speech enhancement method proposed in the embodiment of the present application is used to perform speech enhancement processing on the noisy speech, filter out the noise component in the noisy speech, and thus improve the quality and intelligibility of the speech.

[0073] The noisy speech mentioned above may be a sound signal obtained by collecting a speech signal from a natural environment (an environment containing noise), or may be a noisy speech signal obtained by adding noise to a clean speech signal.

[0074] S102: Input the noisy speech into a pre-trained composite adversarial enhancement network model to obtain a speech enhancement result output by the composite adversarial enhancement network model.

[0075] The speech enhancement method proposed in the embodiment of the present application specifically uses a pre-trained composite adversarial enhancement network model to perform speech enhancement processing on noisy speech, and obtains a speech enhancement result output by the composite adversarial enhancement network model. The speech enhancement result at least includes enhanced speech obtained by speech enhancement of the noisy speech, and the enhanced speech is the speech signal after the noise component is filtered out from the noisy speech.

[0076] As the main innovative content of the embodiments of the present application, the above-mentioned composite adversarial enhancement network model not only has the ability to perform speech enhancement processing on noisy speech to obtain enhanced speech corresponding to the noisy speech, but more importantly, the composite adversarial enhancement network model is trained so that when it performs speech enhancement processing on noisy speech, it can more fully retain the voiceprint information in the speech signal, so that the voiceprint characteristics of the enhanced speech obtained by performing speech enhancement processing on the noisy speech are close to the voiceprint characteristics of the clean speech corresponding to the noisy speech.

[0077] The voiceprint feature of the enhanced speech obtained by performing speech enhancement processing on the noisy speech is close to the voiceprint feature of the clean speech corresponding to the noisy speech. Specifically, the similarity between the voiceprint feature of the enhanced speech obtained by performing speech enhancement processing on the noisy speech and the voiceprint feature of the clean speech corresponding to the noisy speech is greater than a set similarity threshold. In order to enable the composite adversarial enhancement network model to fully retain the voiceprint information in the speech signal during the speech enhancement process, the similarity threshold can be set to a higher threshold, for example, the similarity threshold can be set to 90%.

[0078] In order to make the voiceprint features of the enhanced speech obtained by the above-mentioned composite adversarial enhancement network model through speech enhancement processing on the noisy speech close to the voiceprint features of the clean speech corresponding to the noisy speech, as an exemplary training method, the composite adversarial enhancement network model can be trained in the following way:

[0079] First, the clean speech A is denoised to obtain the noisy speech B, and the above-mentioned composite adversarial enhancement network model is used to perform speech enhancement processing on the above-mentioned noisy speech B to obtain the enhanced speech C.

[0080] Then, the clean speech voiceprint features are extracted from the clean speech A, and the enhanced speech voiceprint features are extracted from the enhanced speech C.

[0081] Finally, the voiceprint feature loss is calculated based on the clean speech voiceprint features and the enhanced speech voiceprint features, and the speech enhancement loss is calculated based on the clean speech A and the enhanced speech C. The voiceprint feature loss and the speech enhancement loss are used as the loss function, and the gradient descent method is used to reversely correct the model parameters.

[0082] The above training process is repeatedly performed until the speech enhancement loss and voiceprint feature loss of the model drop to their respective corresponding threshold ranges, at which point the model training process is terminated.

[0083] In the above training process, the composite adversarial enhancement network model should not only improve the effect of speech enhancement, but also improve the effect of retaining the voiceprint features in the speech, so that the model trained as above can not only achieve satisfactory speech enhancement effect, but also fully retain the voiceprint features in the speech, thereby reducing the speech distortion caused by speech enhancement.

[0084] Based on the above-mentioned compound adversarial enhancement network model obtained through training, when performing speech enhancement on noisy speech, the embodiment of the present application inputs the noisy speech into the compound adversarial enhancement network model to obtain the speech enhancement result output by the compound adversarial enhancement network model. Then, based on the above-mentioned function of the compound adversarial enhancement network model, the enhanced speech in the speech enhancement result obtained not only achieves the purpose of speech enhancement on noisy speech, but also retains more voiceprint information in the noisy speech in the enhanced speech, thereby making the speech distortion of the enhanced speech lower.

[0085] From the above introduction, it can be seen that the speech enhancement method proposed in the embodiment of the present application uses a pre-trained composite adversarial enhancement network model to perform speech enhancement processing on noisy speech to obtain enhanced speech. The composite adversarial enhancement network model has the ability to perform speech enhancement processing on noisy speech to obtain enhanced speech corresponding to the noisy speech, and the voiceprint characteristics of the enhanced speech obtained by the model through speech enhancement processing on noisy speech are close to the voiceprint characteristics of the clean speech corresponding to the noisy speech. Then, by processing noisy speech through the composite adversarial enhancement network model, not only can the purpose of speech enhancement of noisy speech be achieved, but also more voiceprint information in the speech can be retained, thereby reducing the speech distortion caused by speech enhancement.

[0086] Especially when performing speech enhancement on noisy speech with a low signal-to-noise ratio, the technical solution of the embodiment of the present application can more fully retain the voiceprint information in the speech signal while performing speech enhancement on the noisy speech, thereby solving the problem of obvious speech distortion caused by the existing speech enhancement scheme when performing speech enhancement on noisy speech with a low signal-to-noise ratio.

[0087] As a preferred implementation, the speech enhancement result obtained by performing speech enhancement on noisy speech using the above-mentioned composite adversarial enhancement network model includes not only enhanced speech but also audio gain.

[0088] Among them, the audio gain is determined based on the amplitude ratio of the enhanced speech obtained by performing speech enhancement processing on the noisy speech to the noisy speech at each sampling point, that is, the audio gain obtained by the composite adversarial enhancement network model performing speech enhancement processing on the noisy speech represents the amplitude ratio of the speech after speech enhancement to the noisy speech at each sampling point.

[0089] Under normal circumstances, the ratio of clean speech to noisy speech obtained by adding noise to clean speech at each sampling point is basically fixed, that is, the audio gain of clean speech to noisy speech obtained by adding noise to clean speech is basically fixed. Therefore, if the audio gain of the enhanced speech after speech enhancement of noisy speech is close to the audio gain of clean speech relative to noisy speech, it can be said that the enhanced speech after speech enhancement of noisy speech is close to the clean speech, that is, from another aspect, it shows that the speech enhancement effect of noisy speech is better and its speech distortion is lower.

[0090] Based on the above ideas, the embodiment of the present application sets two outputs for the above-mentioned composite adversarial enhancement network model, one is enhanced speech and the other is audio gain.

[0091] During the model training process, the clean speech A is noised to obtain the noisy speech B, and the above-mentioned composite adversarial enhancement network model performs speech enhancement processing on the above-mentioned noisy speech B to obtain the enhanced speech C.

[0092] Then, the clean speech voiceprint feature is extracted from the clean speech A, and the enhanced speech voiceprint feature is extracted from the enhanced speech C. In addition, the audio gain of the clean speech A relative to the noisy speech B is obtained, and the audio gain of the enhanced speech C relative to the noisy speech B is obtained.

[0093] Finally, the voiceprint feature loss is calculated based on the clean voiceprint features and the enhanced voiceprint features, the speech enhancement loss is calculated based on the clean voice A and the enhanced voice C, and the audio gain loss is calculated based on the audio gain of the clean voice A relative to the noisy voice B and the audio gain of the enhanced voice C relative to the noisy voice B. The voiceprint feature loss, speech enhancement loss and audio gain loss are used as the loss function, and the gradient descent method is used to reversely correct the model parameters.

[0094] After the above training, the composite adversarial enhancement network model can not only perform speech enhancement on noisy speech to obtain enhanced speech, but also retain more voiceprint information in the enhanced speech. More importantly, the model can learn the proportional relationship between speech and noise during the training process, which can effectively reduce the mutation degree of enhanced speech and reduce speech distortion, thereby improving the quality of speech listening.

[0095] Since the output of the above-mentioned composite adversarial enhancement network model includes two parts: enhanced speech and audio gain, after obtaining the speech enhancement result output by the model performing speech enhancement processing on the noisy speech, the enhanced speech corresponding to the noisy speech input into the composite adversarial enhancement network model is obtained according to the speech enhancement result.

[0096] Exemplarily, the enhanced speech corresponding to the input noisy speech can be calculated based on the enhanced speech and the audio gain in the speech enhancement result.

[0097] For example, since the audio gain is determined based on the amplitude ratio of the enhanced speech to the noisy speech at each sampling point, the enhanced speech can be calculated based on the noisy speech and the audio gain output by the model. Then, the calculated enhanced speech is weighted summed with the enhanced speech output by the composite adversarial enhancement network model to obtain the enhanced speech corresponding to the input noisy speech.

[0098] As an optional implementation, the composite adversarial enhancement network model in the embodiment of the present application is composed of at least two deep adversarial enhancement modules connected in series in sequence. Each deep adversarial enhancement module is used to perform speech enhancement processing and output enhanced speech obtained by the speech enhancement processing.

[0099] For example, see Figure 2 As shown, the above-mentioned composite adversarial enhancement network model in the embodiment of the present application is specifically composed of three deep adversarial enhancement modules connected in series in sequence. According to the cascade relationship formed by the three modules connected in series, they can be named as the first deep adversarial enhancement module, the second deep adversarial enhancement module and the third deep adversarial enhancement module from the front stage to the back stage respectively.

[0100] See also Figure 3 As shown in the figure, each deep adversarial enhancement module consists of 3 layers of one-dimensional convolution and 1 layer of one-dimensional deconvolution. The convolution kernel sizes are (32, 1, 1, 16) and the step sizes are (16, 1, 1, 8). The ReLU activation function and InstanceNorm regularization are used.

[0101] At the same time, each deep adversarial enhancement module contains three multi-scale residual units, such as Figure 4 As shown in the figure, each multi-scale residual unit consists of three one-dimensional convolutions, InstanceNorm regularization and PReLU activation functions. The step size of the three one-dimensional convolutions is 1, and the convolution kernel sizes are (1, 8*2 n ,1), that is, the size of the convolution kernel of the second convolution is variable, recorded as variable convolution. The multi-scale residual unit adopts a residual structure to superimpose the output and the input of this unit, which is conducive to retaining the gradient information of the data when settling in the deep network.

[0102] Each deep adversarial enhancement module outputs an enhanced speech output, wherein the third deep adversarial enhancement module not only outputs an enhanced speech, but also uses another one-dimensional convolution operation and decoding step to obtain a predicted audio gain output3_gain.

[0103] Based on the above model structure, when noisy speech is input into the above-mentioned composite adversarial enhancement network model, the noisy speech is input into the first deep adversarial enhancement module, and the enhanced speech output1 output by the first deep adversarial enhancement module is fused with the noisy speech and input into the second deep adversarial enhancement module; the enhanced speech output2 output by the second deep adversarial enhancement module is fused with the noisy speech and input into the third deep adversarial enhancement module; the third deep adversarial enhancement module outputs enhanced speech output3 and audio gain output3_gain as the speech enhancement result output by the composite adversarial enhancement network model.

[0104] Furthermore, the composite adversarial enhancement network model is trained so that the difference between the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module and the signal-to-noise ratio of the input speech is the same as the difference between the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module and the signal-to-noise ratio of the input speech. That is, the first deep adversarial enhancement module and the second deep adversarial enhancement module can improve the signal-to-noise ratio of the input speech by the same value. For example Figure 2 As shown in , after the noisy speech is processed by the first deep adversarial enhancement module, the signal-to-noise ratio is improved by 10dB, and after being processed by the second deep adversarial enhancement module, the signal-to-noise ratio is improved by another 10dB. Based on the design of the multi-level deep adversarial enhancement module, the model can achieve a gradual improvement in the signal-to-noise ratio of noisy speech, improve the model's robustness to unknown noise, and reduce speech distortion caused by speech enhancement.

[0105] Based on the design of the above-mentioned multi-level deep adversarial enhancement module, the enhanced speech output by the above-mentioned composite adversarial enhancement network model is actually composed of the enhanced speech output by each deep adversarial enhancement module.

[0106] Therefore, the above-mentioned speech enhancement result outputted by the composite adversarial enhancement network model is used to obtain the enhanced speech corresponding to the input noisy speech, which can be specifically achieved by executing the following steps A1-A3:

[0107] A1. Perform weighted summation on the enhanced speech output by each deep adversarial enhancement module to obtain a first enhancement result.

[0108] For example, suppose Figure 2The enhanced speech output by each deep adversarial enhancement module of the composite adversarial enhancement network model is output1, output2, and output3 respectively, and the first enhancement result enh_1 can be calculated by the following formula:

[0109] enh_1=0.3*output1+0.3*output2+0.4*output3

[0110] A2. Calculate a second enhancement result according to the noisy speech and the audio gain.

[0111] For example, assuming that the audio gain output by the composite adversarial enhancement network model is output3_gain, the second enhancement result enh_2 can be calculated by the following formula:

[0112] enh_2=N_way+logoutput3_gain

[0113] Wherein, represents the original sampling data of noisy speech.

[0114] A3. Calculate and obtain enhanced speech corresponding to the input noisy speech according to the first enhancement result and the second enhancement result.

[0115] Specifically, the enhanced speech EnH corresponding to the input noisy speech is calculated according to the following formula:

[0116] EnH=0.5*enh_1+0.5*enh_2

[0117] That is, the enhanced speech EnH is the final enhanced result of speech enhancement on the noisy speech input into the composite adversarial enhancement network model.

[0118] The training process of the above-mentioned composite adversarial enhanced network model is introduced below.

[0119] Specifically, the training of the above-mentioned composite adversarial enhanced network model can be carried out according to Figure 5 The process shown is implemented by executing the following steps B1-B5:

[0120] B1. Obtain a noisy speech sample, a clean speech corresponding to the noisy speech sample, and determine a target audio gain based on the noisy speech sample and the corresponding clean speech.

[0121] Specifically, the noisy speech is obtained by adding noise to the clean speech input_clean, and the noisy speech can be used as the noisy speech sample input for model training.

[0122] According to the amplitude of the clean speech input_clean and the noisy speech sample input, the target audio gain label_gain can be calculated according to the following formula:

[0123]

[0124] B2. Input the noisy speech sample into the composite adversarial enhancement network model to obtain enhanced speech and audio gain output by the composite adversarial enhancement network model.

[0125] Specifically, the noisy speech sample input is input into the composite adversarial enhancement network model to obtain the enhanced speech output and the audio gain output_gain.

[0126] B3. Acquire the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model, and acquire the voiceprint features of the clean speech corresponding to the noisy speech sample.

[0127] Voiceprint features are extracted for the enhanced speech output output by the composite adversarial enhancement network model and the clean speech input_clean corresponding to the noisy speech sample input, respectively, to obtain the voiceprint features of the enhanced speech output and the voiceprint features of the clean speech input_clean.

[0128] Exemplarily, the enhanced speech output output by the composite adversarial enhancement network model and the clean speech input_clean corresponding to the noisy speech sample input are respectively input into a pre-trained voiceprint feature extraction network to obtain the voiceprint features of the enhanced speech output output by the composite adversarial enhancement network model and the voiceprint features of the clean speech input_clean corresponding to the noisy speech sample input;

[0129] The voiceprint feature extraction network is obtained by extracting the speaker's voiceprint features from the input speech and performing speaker recognition training.

[0130] Specifically, assuming there are N speakers, a time-delay deep neural network (TDNN) is used to train the voiceprint recognition task, and the voiceprint recognition training is performed based on the cross entropy loss function as the training criterion. After completing the voiceprint recognition training, the output of the fully connected layer after the pooling layer of the network is used as the voiceprint feature. At this time, the network can be used as a voiceprint feature extraction network.

[0131] In the training process of the above-mentioned voiceprint feature extraction network, since classification criteria are used for training, the extracted voiceprint features have a strong voiceprint representation ability, that is, they have a better ability to distinguish high-frequency information in the speech signal.

[0132] B4. Determine the audio mapping loss based on the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample, determine the audio gain loss based on the audio gain output by the composite adversarial enhancement network model and the target audio gain, and determine the voiceprint feature loss based on the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and the voiceprint features of the clean speech corresponding to the noisy speech sample.

[0133] Specifically, the L1 norm between the enhanced speech output output by the composite adversarial enhancement network model and the clean speech input_clean corresponding to the noisy speech sample input is calculated as the audio mapping loss Loss_G way .

[0134] According to the following formula, the mean square error of the audio gain output_gain output by the composite adversarial enhancement network model is calculated as the audio gain loss Loss_G gain :

[0135]

[0136] Where N represents the total number of speakers in the speech.

[0137] According to the following formula, the cosine distance between the voiceprint feature of the enhanced speech output output by the composite adversarial enhancement network model and the voiceprint feature of the clean speech input_clean corresponding to the noisy speech sample input is calculated as the voiceprint feature loss Loss vid :

[0138]

[0139] Where N is the total number of speakers in the speech; v i Indicates the voiceprint features of the clean speech input_clean; v i 'Indicates the voiceprint features for enhanced speech output.

[0140] B5. Perform parameter correction on the composite adversarial enhancement network model at least according to the audio mapping loss, the audio gain loss and the voiceprint feature loss.

[0141] Specifically, the gradient descent method is used to correct the parameters of the above-mentioned composite adversarial enhancement network model, so that the audio mapping loss, audio gain loss and voiceprint feature loss of the model are reduced respectively.

[0142] Repeat the training process of B1-B5 above until all losses of the model drop to the corresponding numerical range, and then end the model training process.

[0143] Further, based on the model training process shown in B1-B5 above, the embodiment of the present application performs the following step B6 while performing step B4:

[0144] B6. Determine the speech difference loss by comparing the enhanced speech output by the composite adversarial enhancement network model with the clean speech corresponding to the noisy speech sample.

[0145] Specifically, the enhanced speech output output by the composite adversarial enhancement network model is compared with the clean speech input_clean corresponding to the noisy speech sample input at the speech level to determine the speech difference loss between the two.

[0146] As an exemplary implementation method, the embodiment of the present application uses a pre-trained speech discrimination network to discriminate the enhanced speech output output by the composite adversarial enhancement network model and the clean speech input_clean corresponding to the noisy speech sample input; and determines the speech difference loss according to the discrimination result output by the speech discrimination network;

[0147] The speech discrimination network has the ability to distinguish between noisy speech and clean speech corresponding to the noisy speech.

[0148] The speech discrimination network judges the clean samples and determines whether the speech is "true" or "false". "True" means that the sample is clean speech, and "false" means that the speech is the speech output by the composite adversarial enhancement network model. The class label of the clean sample is 1, and the class label of the enhanced speech is 0. The L2 norm is used as the speech discrimination network loss function. The loss function is divided into two parts. First, for clean speech, where x i represents the real clean speech, D(x i ) is the output of the speech discrimination network, and N is the number of samples. Secondly, for the enhanced speech x' output by the composite adversarial enhancement network model, x' is sent to the speech discrimination network for category determination. The loss of the speech discrimination network Represents the enhanced speech output by the composite adversarial enhancement network model. The total loss of the speech discrimination network is:

[0149] By continuously training the speech discrimination network, its ability to distinguish clean speech and enhanced speech is improved.

[0150] Based on the speech discrimination network obtained by the above training, in the training process of the above-mentioned composite adversarial enhancement network model, the speech discrimination network is used to discriminate the enhanced speech output output by the composite adversarial enhancement network model and the clean speech input_clean corresponding to the noisy speech sample input, and based on the discrimination result of the speech discrimination network, the speech difference loss Loss is calculated according to the following formula G_D :

[0151]

[0152] That is, the various loss functions calculated in the above steps B4 and B6, when executing step B5, specifically according to the audio mapping loss, the audio gain loss, the voiceprint feature loss and the speech difference loss, the parameters of the composite adversarial enhancement network model are corrected.

[0153] Specifically, the gradient descent method is used to correct the parameters of the above-mentioned composite adversarial enhancement network model, so that the audio mapping loss, audio gain loss, voiceprint feature loss and speech difference loss of the model are reduced respectively.

[0154] In the above training process, the differences between the enhanced speech output by the composite adversarial enhancement network model and the clean speech in terms of audio gain, voiceprint characteristics, speech similarity, etc. are taken into consideration at the same time, and the parameters of the composite adversarial enhancement network model are corrected based on these differences. This ensures that the trained composite adversarial enhancement network model can make the enhanced speech closer to the clean speech and retain more voiceprint information when performing speech enhancement.

[0155] For further information, see Figure 2 As shown, the above-mentioned composite adversarial enhancement network model is specifically composed of a first deep adversarial enhancement module, a second deep adversarial enhancement module and a third deep adversarial enhancement module connected in series. Among them, the noisy speech input into the composite adversarial enhancement network model is input into the first deep adversarial enhancement module, and the output of the third deep adversarial enhancement module is used as the speech enhancement result output by the composite adversarial enhancement network model.

[0156] In order to make the difference between the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module and the signal-to-noise ratio of the input speech the same as the difference between the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module and the signal-to-noise ratio of the input speech.

[0157] The embodiment of the present application sets that when the parameters of the composite adversarial enhancement network model are corrected according to various loss functions, not only the gradient descent of each loss function is taken as the goal, but also the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module is increased by a first preset value relative to the signal-to-noise ratio of its input speech, and the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module is increased by the first preset value relative to the signal-to-noise ratio of its input speech. After the above training, the composite adversarial enhancement network model can gradually improve the signal-to-noise ratio of noisy speech, improve the robustness to unknown noise, and reduce speech distortion.

[0158] Based on the training process of the composite adversarial enhancement network model described above in the embodiment of the present application, it can be understood that the embodiment of the present application introduces voiceprint information to guide the speech enhancement process of the model, and through voiceprint constraints, the model's ability to protect high-frequency signal components is improved, thereby reducing speech distortion. At the same time, a multi-scale residual unit is designed in the composite adversarial enhancement network model, which can improve the utilization of multi-scale spatial information, and combine progressive learning and multi-task learning methods to further improve the sense of speech and reduce speech distortion. In addition, the speech enhancement method based on the composite adversarial enhancement network model can achieve end-to-end speech enhancement, thereby reducing processing flows such as feature extraction, which is convenient for engineering implementation.

[0159] Corresponding to the above-mentioned speech enhancement method, the present application embodiment also proposes a speech enhancement device, see Figure 6 As shown, the device at least includes:

[0160] The speech enhancement unit 100 is used to input the noisy speech into the pre-trained composite adversarial enhancement network model to obtain the speech enhancement result output by the composite adversarial enhancement network model, wherein the speech enhancement result at least includes the enhanced speech;

[0161] Among them, the composite adversarial enhancement network model has the ability to perform speech enhancement processing on noisy speech to obtain enhanced speech corresponding to the noisy speech, and the voiceprint characteristics of the enhanced speech obtained by the model through speech enhancement processing on noisy speech are close to the voiceprint characteristics of the clean speech corresponding to the noisy speech.

[0162] Optionally, the speech enhancement result also includes an audio gain, and the audio gain is determined based on an amplitude ratio of the enhanced speech obtained by performing speech enhancement processing on the noisy speech to the noisy speech at each sampling point;

[0163] The audio gain obtained by the composite adversarial enhancement network model performing speech enhancement processing on the noisy speech is close to the audio gain of the clean speech corresponding to the noisy speech relative to the noisy speech;

[0164] The device also includes:

[0165] The speech processing unit is used to obtain enhanced speech corresponding to the input noisy speech according to the speech enhancement result.

[0166] Optionally, acquiring, according to the speech enhancement result, an enhanced speech corresponding to the input noisy speech includes:

[0167] According to the enhanced speech and the audio gain in the speech enhancement result, the enhanced speech corresponding to the input noisy speech is calculated.

[0168] Optionally, the composite adversarial enhancement network model includes at least two deep adversarial enhancement modules, the deep adversarial enhancement modules are connected in series in sequence, each deep adversarial enhancement module is used to perform speech enhancement processing, and outputs enhanced speech obtained by the speech enhancement processing.

[0169] Optionally, the composite adversarial enhancement network includes a first deep adversarial enhancement module, a second deep adversarial enhancement module and a third deep adversarial enhancement module connected in series in sequence;

[0170] Among them, the noisy speech input into the composite adversarial enhancement network model is input into the first deep adversarial enhancement module, and the output of the third deep adversarial enhancement module is used as the speech enhancement result output by the composite adversarial enhancement network model; and the difference between the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module and the signal-to-noise ratio of the input speech is the same as the difference between the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module and the signal-to-noise ratio of the input speech.

[0171] Optionally, the first depth adversarial enhancement module, the second depth adversarial enhancement module and the third depth adversarial enhancement module respectively include a set number of multi-scale residual units.

[0172] Optionally, acquiring, according to the speech enhancement result, an enhanced speech corresponding to the input noisy speech includes:

[0173] Performing weighted summation on the enhanced speech output by each deep adversarial enhancement module to obtain a first enhancement result;

[0174] Calculating a second enhancement result according to the noisy speech and the audio gain;

[0175] According to the first enhancement result and the second enhancement result, an enhanced speech corresponding to the input noisy speech is calculated.

[0176] Optionally, the training process of the composite adversarial enhanced network model includes:

[0177] Acquire a noisy speech sample, a clean speech corresponding to the noisy speech sample, and a target audio gain determined based on the noisy speech sample and the corresponding clean speech;

[0178] Inputting the noisy speech sample into the composite adversarial enhancement network model to obtain enhanced speech and audio gain output by the composite adversarial enhancement network model;

[0179] Acquire the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model, and acquire the voiceprint features of the clean speech corresponding to the noisy speech sample;

[0180] Determine an audio mapping loss according to the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample, determine an audio gain loss according to the audio gain output by the composite adversarial enhancement network model and the target audio gain, and determine a voiceprint feature loss according to the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and the voiceprint features of the clean speech corresponding to the noisy speech sample;

[0181] Parameters of the composite adversarial enhancement network model are corrected at least according to the audio mapping loss, the audio gain loss and the voiceprint feature loss.

[0182] Optionally, the training process of the composite adversarial enhanced network model further includes:

[0183] Determining the speech difference loss by comparing the enhanced speech output by the composite adversarial enhancement network model with the clean speech corresponding to the noisy speech sample;

[0184] The step of performing parameter correction on the composite adversarial enhancement network model at least according to the audio mapping loss, the audio gain loss, and the voiceprint feature loss includes:

[0185] According to the audio mapping loss, the audio gain loss, the voiceprint feature loss and the speech difference loss, parameters of the composite adversarial enhancement network model are corrected.

[0186] Optionally, the obtaining of the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model, and the obtaining of the voiceprint features of the clean speech corresponding to the noisy speech sample, include:

[0187] Inputting the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample into a pre-trained voiceprint feature extraction network respectively, to obtain the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and the voiceprint features of the clean speech corresponding to the noisy speech sample;

[0188] The voiceprint feature extraction network is obtained by extracting the speaker's voiceprint features from the input speech and performing speaker recognition training.

[0189] Optionally, determining the speech difference loss by comparing the enhanced speech output by the composite adversarial enhancement network model with the clean speech corresponding to the noisy speech sample includes:

[0190] Using a pre-trained speech discrimination network, the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample are discriminated;

[0191] Determining the speech difference loss according to the discrimination result output by the speech discrimination network;

[0192] The speech discrimination network has the ability to distinguish between noisy speech and clean speech corresponding to the noisy speech.

[0193] Optionally, the composite adversarial enhancement network includes a first deep adversarial enhancement module, a second deep adversarial enhancement module and a third deep adversarial enhancement module connected in series in sequence; wherein the noisy speech input into the composite adversarial enhancement network model is input into the first deep adversarial enhancement module, and the output of the third deep adversarial enhancement module is used as the speech enhancement result output by the composite adversarial enhancement network model;

[0194] The parameter correction of the composite adversarial enhancement network model includes:

[0195] The parameters of the composite adversarial enhancement network model are corrected with the goal of increasing the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module by a first preset value relative to the signal-to-noise ratio of its input speech, and increasing the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module by the first preset value relative to the signal-to-noise ratio of its input speech.

[0196] Specifically, for the specific structure and corresponding processing content of the above-mentioned speech enhancement device, please refer to the content of the above-mentioned method embodiment, which will not be repeated here.

[0197] Another embodiment of the present application also provides a voice enhancement device, see Figure 7 As shown, the device includes:

[0198] Memory 200 and processor 210;

[0199] The memory 200 is connected to the processor 210 and is used to store programs;

[0200] The processor 210 is used to implement the speech enhancement method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0201] Specifically, the above-mentioned speech enhancement device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0202] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0203] A bus may include a pathway that transfers information between components of a computer system.

[0204] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0205] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0206] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0207] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0208] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0209] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0210] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of the speech enhancement method provided in the embodiment of the present application.

[0211] Another embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the speech enhancement method provided in any of the above embodiments is implemented.

[0212] Specifically, the specific working contents of each part of the above-mentioned speech enhancement device, as well as the specific processing contents of the computer program on the above-mentioned storage medium when being executed by the processor, can all be found in the contents of each embodiment of the above-mentioned speech enhancement method, and will not be repeated here.

[0213] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0214] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0215] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0216] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0217] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0218] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0220] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0221] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0222] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0223] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech enhancement method, characterized in that: include: Inputting noisy speech into a pre-trained composite adversarial enhancement network model to obtain a speech enhancement result output by the composite adversarial enhancement network model, wherein the speech enhancement result at least includes enhanced speech; The voiceprint feature of the enhanced speech obtained by the composite adversarial enhancement network model through speech enhancement processing on the noisy speech is close to the voiceprint feature of the clean speech corresponding to the noisy speech; the voiceprint feature of the enhanced speech obtained through speech enhancement processing on the noisy speech is close to the voiceprint feature of the clean speech corresponding to the noisy speech, specifically, the similarity between the voiceprint feature of the enhanced speech obtained through speech enhancement processing on the noisy speech and the voiceprint feature of the clean speech corresponding to the noisy speech is greater than a set similarity threshold; The training process of the composite adversarial enhanced network model includes: Acquire a noisy speech sample, a clean speech corresponding to the noisy speech sample, and a target audio gain determined based on the noisy speech sample and the corresponding clean speech; Inputting the noisy speech sample into the composite adversarial enhancement network model to obtain enhanced speech and audio gain output by the composite adversarial enhancement network model; Acquire the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model, and acquire the voiceprint features of the clean speech corresponding to the noisy speech sample; Determine an audio mapping loss according to the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample, determine an audio gain loss according to the audio gain output by the composite adversarial enhancement network model and the target audio gain, and determine a voiceprint feature loss according to the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and the voiceprint features of the clean speech corresponding to the noisy speech sample; Parameters of the composite adversarial enhancement network model are corrected at least according to the audio mapping loss, the audio gain loss and the voiceprint feature loss.

2. The method according to claim 1, characterized in that: The speech enhancement result also includes an audio gain, which is determined based on the amplitude ratio of the enhanced speech obtained by performing speech enhancement processing on the noisy speech to the noisy speech at each sampling point; The audio gain obtained by the composite adversarial enhancement network model performing speech enhancement processing on the noisy speech is close to the audio gain of the clean speech corresponding to the noisy speech relative to the noisy speech; The method further comprises: According to the speech enhancement result, an enhanced speech corresponding to the input noisy speech is obtained.

3. The method according to claim 2, characterized in that The step of acquiring, according to the speech enhancement result, an enhanced speech corresponding to the input noisy speech comprises: According to the enhanced speech and the audio gain in the speech enhancement result, the enhanced speech corresponding to the input noisy speech is calculated.

4. The method according to claim 2, characterized in that: The composite adversarial enhancement network model includes at least two deep adversarial enhancement modules, each of which is connected in series in sequence. Each deep adversarial enhancement module is used to perform speech enhancement processing and output enhanced speech obtained by the speech enhancement processing.

5. The method according to claim 4, characterized in that The composite adversarial enhancement network comprises a first deep adversarial enhancement module, a second deep adversarial enhancement module and a third deep adversarial enhancement module connected in series in sequence; Among them, the noisy speech input into the composite adversarial enhancement network model is input into the first deep adversarial enhancement module, and the output of the third deep adversarial enhancement module is used as the speech enhancement result output by the composite adversarial enhancement network model; and the difference between the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module and the signal-to-noise ratio of the input speech is the same as the difference between the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module and the signal-to-noise ratio of the input speech.

6. The method according to claim 5, characterized in that The first deep adversarial enhancement module, the second deep adversarial enhancement module and the third deep adversarial enhancement module respectively include a set number of multi-scale residual units.

7. The method according to claim 4, characterized in that The step of acquiring, according to the speech enhancement result, an enhanced speech corresponding to the input noisy speech comprises: Performing weighted summation on the enhanced speech output by each deep adversarial enhancement module to obtain a first enhancement result; Calculating a second enhancement result according to the noisy speech and the audio gain; According to the first enhancement result and the second enhancement result, an enhanced speech corresponding to the input noisy speech is calculated.

8. The method according to claim 1, characterized in that The method further comprises: Determining the speech difference loss by comparing the enhanced speech output by the composite adversarial enhancement network model with the clean speech corresponding to the noisy speech sample; The step of performing parameter correction on the composite adversarial enhancement network model at least according to the audio mapping loss, the audio gain loss, and the voiceprint feature loss includes: According to the audio mapping loss, the audio gain loss, the voiceprint feature loss and the speech difference loss, parameters of the composite adversarial enhancement network model are corrected.

9. The method according to claim 1, characterized in that: The step of obtaining the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and obtaining the voiceprint features of the clean speech corresponding to the noisy speech sample includes: Inputting the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample into a pre-trained voiceprint feature extraction network respectively, to obtain the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and the voiceprint features of the clean speech corresponding to the noisy speech sample; The voiceprint feature extraction network is obtained by extracting the speaker's voiceprint features from the input speech and performing speaker recognition training.

10. The method according to claim 8, characterized in that The step of comparing the enhanced speech output by the composite adversarial enhancement network model with the clean speech corresponding to the noisy speech sample to determine the speech difference loss includes: Using a pre-trained speech discrimination network, the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample are discriminated; Determining the speech difference loss according to the discrimination result output by the speech discrimination network; The speech discrimination network has the ability to distinguish between noisy speech and clean speech corresponding to the noisy speech.

11. The method according to claim 1, characterized in that: The composite adversarial enhancement network includes a first deep adversarial enhancement module, a second deep adversarial enhancement module and a third deep adversarial enhancement module which are connected in series in sequence; wherein the noisy speech input into the composite adversarial enhancement network model is input into the first deep adversarial enhancement module, and the output of the third deep adversarial enhancement module is used as the speech enhancement result output by the composite adversarial enhancement network model; The parameter correction of the composite adversarial enhancement network model includes: The parameters of the composite adversarial enhancement network model are corrected with the goal of increasing the signal-to-noise ratio of the output speech of the first deep adversarial enhancement module by a first preset value relative to the signal-to-noise ratio of its input speech, and increasing the signal-to-noise ratio of the output speech of the second deep adversarial enhancement module by the first preset value relative to the signal-to-noise ratio of its input speech.

12. A speech enhancement device, characterized in that: include: A speech enhancement unit, used for inputting noisy speech into a pre-trained composite adversarial enhancement network model to obtain a speech enhancement result output by the composite adversarial enhancement network model, wherein the speech enhancement result at least includes enhanced speech; The voiceprint features of the enhanced speech obtained by the composite adversarial enhancement network model performing speech enhancement processing on the noisy speech are close to the voiceprint features of the clean speech corresponding to the noisy speech; The voiceprint feature of the enhanced speech obtained by performing speech enhancement processing on the noisy speech is close to the voiceprint feature of the clean speech corresponding to the noisy speech, and specifically, the similarity between the voiceprint feature of the enhanced speech obtained by performing speech enhancement processing on the noisy speech and the voiceprint feature of the clean speech corresponding to the noisy speech is greater than a set similarity threshold; The training process of the composite adversarial enhanced network model includes: Acquire a noisy speech sample, a clean speech corresponding to the noisy speech sample, and a target audio gain determined based on the noisy speech sample and the corresponding clean speech; Inputting the noisy speech sample into the composite adversarial enhancement network model to obtain enhanced speech and audio gain output by the composite adversarial enhancement network model; Acquire the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model, and acquire the voiceprint features of the clean speech corresponding to the noisy speech sample; Determine an audio mapping loss according to the enhanced speech output by the composite adversarial enhancement network model and the clean speech corresponding to the noisy speech sample, determine an audio gain loss according to the audio gain output by the composite adversarial enhancement network model and the target audio gain, and determine a voiceprint feature loss according to the voiceprint features of the enhanced speech output by the composite adversarial enhancement network model and the voiceprint features of the clean speech corresponding to the noisy speech sample; Parameters of the composite adversarial enhancement network model are corrected at least according to the audio mapping loss, the audio gain loss and the voiceprint feature loss.

13. A speech enhancement device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the speech enhancement method as described in any one of claims 1 to 11 by running the program in the memory.

14. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the speech enhancement method according to any one of claims 1 to 11 is implemented.

Images