A speech enhancement method, apparatus and device

Abstract

This application provides a speech enhancement method, apparatus, and device, relating to the field of data processing technology. The method includes: acquiring noisy speech to be enhanced; converting the speech to be enhanced into a noisy context representation by inputting the speech to be enhanced into a self-supervised module of a speech enhancement model; and converting the noisy context representation into enhanced speech by inputting it into a vocoder module of the speech enhancement model, resulting in clean speech. Therefore, the speech enhancement method provided by this application can enhance the noisy speech to be enhanced, effectively suppressing noise and reverberation in the speech to be enhanced, and significantly improving the naturalness and clarity of the enhanced speech (clean speech).

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a speech enhancement method, apparatus, and device. Background Technology

[0002] Voice enhancement technology refers to techniques that improve the clarity and naturalness of speech by suppressing background noise and reverberation. In applications such as voice interaction in smart homes and voice calls on mobile devices in noisy environments, the performance of voice enhancement technology directly impacts the user experience.

[0003] In related technologies, speech enhancement is typically performed using speech enhancement models based on deep neural networks (DNNs). Speech enhancement models are mainly divided into two categories: generative models and discriminative models. Generative models learn the probability distribution of clean speech and reconstruct noisy speech signals that conform to this probability distribution. However, generative models suffer from mode collapse during training, meaning they may overemphasize certain speech features while ignoring others, resulting in unnatural-looking generated speech signals. Discriminative models, on the other hand, suppress noise by minimizing the difference between the training speech output by the noisy enhancement model and the corresponding clean speech after inputting noisy speech. However, discriminative models rely on a large amount of "noisy speech - clean speech" data for training. When faced with unseen noise types, the discriminative model lacks the ability to learn these unseen noise types, making it difficult to effectively improve speech clarity.

[0004] Therefore, how to enhance noisy speech to obtain cleaner speech with higher naturalness and clarity has become an urgent technical problem to be solved. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a speech enhancement method, apparatus, and device that can enhance noisy speech to obtain cleaner speech with higher naturalness and clarity.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In a first aspect, this application discloses a speech enhancement method, the method comprising:

[0008] Acquire the speech to be enhanced, wherein the speech to be enhanced is noisy speech;

[0009] By inputting the speech to be enhanced into the self-supervised module of the speech enhancement model, the speech to be enhanced is converted into a noisy contextual representation;

[0010] By inputting the noisy context representation into the vocoder module of the speech enhancement model, the noisy context representation is converted into enhanced speech, which is clean speech.

[0011] Optionally, the self-supervised module includes a feature encoder and a Transformer module; the step of converting the speech to be enhanced into a noisy context representation includes:

[0012] The feature encoder is used to extract features from the speech to be enhanced to obtain noisy features;

[0013] The noisy features are randomly masked using the Transformer module, and a noisy context representation is obtained based on the masked noisy features.

[0014] Optionally, the vocoder module includes a generator, which includes a multi-layer one-dimensional transposed convolution and a multi-resolution fusion module; the step of converting the noisy context representation into enhanced speech includes:

[0015] The noisy context is upsampled by the multi-layer one-dimensional transposed convolution to obtain intermediate features at multiple time scales, and the temporal resolution of each intermediate feature is aligned with the temporal resolution of the speech to be enhanced.

[0016] The multi-resolution fusion module fuses intermediate features from multiple time scales to obtain enhanced speech.

[0017] Optionally, the vocoder module further includes a discriminator, which includes a multi-scale discriminator and a multi-period discriminator; the step of fusing intermediate features from the multiple time scales through the multi-resolution fusion module to obtain enhanced speech includes:

[0018] The multi-resolution fusion module fuses intermediate features from multiple time scales to obtain fused speech.

[0019] The multi-scale discriminator performs level evaluation of the fused speech at multiple resolution levels, and the multi-period discriminator performs component evaluation of the periodic components of the fused speech.

[0020] When both the level assessment and the component assessment meet the corresponding assessment conditions, the fused speech is determined to be enhanced speech.

[0021] Optionally, the training method of the self-supervised module is as follows:

[0022] Acquire noisy and clean speech samples;

[0023] The feature encoder is used to extract features from the noisy speech sample and the clean speech sample, respectively, to obtain noisy speech sample features and clean speech sample features.

[0024] The Transformer module randomly masks the noisy features and clean features of the sample, and obtains the noisy context representation and clean context representation of the sample based on the masked noisy features and clean features, respectively.

[0025] The self-supervised module is trained based on the noisy context representation of the sample, the clean context representation of the sample, the noisy features of the sample, the clean features of the sample, and the total loss function, wherein the total loss function includes a contrastive loss function and a consistency loss function.

[0026] Optionally, the training method for the vocoder module is as follows:

[0027] The sample noisy context is upsampled by the multi-layer one-dimensional transposed convolution to obtain sample intermediate features at multiple time scales. The temporal resolution of each sample intermediate feature is aligned with the temporal resolution of the sample noisy speech.

[0028] The multi-resolution fusion module fuses intermediate features of samples from multiple time scales to obtain sample-enhanced speech.

[0029] The generator of the vocoder module is trained based on the clean speech sample, the enhanced speech sample, and the generator's loss function. The generator's loss function includes a first adversarial loss function, a feature matching loss function, a Mel-spectrum loss function, an amplitude loss function, and a complex loss function.

[0030] The discriminator of the vocoder module is trained based on the clean speech samples, the enhanced speech samples, and the second adversarial loss function.

[0031] Secondly, this application discloses a voice enhancement device, the device comprising: a voice acquisition module, a first conversion module, and a second conversion module;

[0032] The speech acquisition module is used to acquire the speech to be enhanced, which is noisy speech;

[0033] The first conversion module is used to convert the speech to be enhanced into a noisy contextual representation by inputting the speech to be enhanced into a self-supervised module of the speech enhancement model;

[0034] The second conversion module is used to convert the noisy context representation into enhanced speech by inputting the noisy context representation into the vocoder module of the speech enhancement model, wherein the enhanced speech is clean speech.

[0035] Optionally, the self-supervised module includes a feature encoder and a Transformer module; the first transformation module includes: a first transformation submodule and a second transformation submodule.

[0036] The first conversion submodule is used to extract features from the speech to be enhanced through the feature encoder to obtain noisy features;

[0037] The second transformation submodule is used to randomly mask the noisy features through the Transformer module, and obtain a noisy context representation based on the masked noisy features.

[0038] Optionally, the vocoder module includes a generator, which includes a multi-layer one-dimensional transposed convolution and a multi-resolution fusion module; the second conversion module includes a third conversion submodule and a fourth conversion submodule;

[0039] The third transformation submodule is used to upsample the noisy context through the multi-layer one-dimensional transposed convolution to obtain intermediate features at multiple time scales, and the temporal resolution of each intermediate feature is aligned with the temporal resolution of the speech to be enhanced.

[0040] The fourth conversion submodule is used to fuse intermediate features from multiple time scales through the multi-resolution fusion module to obtain enhanced speech.

[0041] Optionally, the vocoder module further includes a discriminator, which includes a multi-scale discriminator and a multi-period discriminator; the fourth conversion submodule is specifically used for:

[0042] The multi-resolution fusion module fuses intermediate features from multiple time scales to obtain fused speech.

[0043] The multi-scale discriminator performs level evaluation of the fused speech at multiple resolution levels, and the multi-period discriminator performs component evaluation of the periodic components of the fused speech.

[0044] When both the level assessment and the component assessment meet the corresponding assessment conditions, the fused speech is determined to be enhanced speech.

[0045] Optionally, the training unit of the self-supervised module is specifically as follows:

[0046] The first training unit is used to acquire noisy speech samples and clean speech samples.

[0047] The second training unit is used to extract features from the noisy speech sample and the clean speech sample respectively through the feature encoder to obtain noisy features and clean features of the sample.

[0048] The third training unit is used to randomly mask the noisy features and clean features of the sample through the Transformer module, and to obtain the noisy context representation and the clean context representation of the sample based on the masked noisy features and the masked clean features, respectively.

[0049] The fourth training unit is used to train the self-supervised module based on the noisy context representation of the sample, the clean context representation of the sample, the noisy features of the sample, the clean features of the sample, and the total loss function, wherein the total loss function includes a contrastive loss function and a consistency loss function.

[0050] Optionally, the training unit of the vocoder module is specifically as follows:

[0051] The fifth training unit is used to upsample the noisy context of the sample through the multi-layer one-dimensional transposed convolution to obtain intermediate features of the sample at multiple time scales, and the temporal resolution of each intermediate feature is aligned with the temporal resolution of the noisy speech sample.

[0052] The sixth training unit is used to fuse intermediate features of samples from multiple time scales through the multi-resolution fusion module to obtain sample-enhanced speech.

[0053] The seventh training unit is used to train the generator of the vocoder module based on the clean speech sample, the enhanced speech sample, and the generator's loss function. The generator's loss function includes a first adversarial loss function, a feature matching loss function, a Mel-spectrum loss function, an amplitude loss function, and a complex loss function.

[0054] The eighth training unit is used to train the discriminator of the vocoder module based on the clean speech samples, the enhanced speech samples, and the second adversarial loss function.

[0055] Thirdly, this application discloses a voice enhancement device, the device comprising: a memory and a processor;

[0056] The memory is used to store programs;

[0057] The processor is configured to execute the program to implement the various steps of the speech enhancement method as described in the first aspect.

[0058] Compared with the prior art, this application has the following beneficial effects:

[0059] This application provides a speech enhancement method, apparatus, and device. The method includes: acquiring noisy speech to be enhanced; converting the speech to be enhanced into a noisy context representation by inputting the speech to be enhanced into a self-supervised module of a speech enhancement model; and converting the noisy context representation into enhanced speech by inputting the noisy context representation into a vocoder module of the speech enhancement model, resulting in clean speech. Therefore, the speech enhancement method provided by this application can enhance the noisy speech to be enhanced, effectively suppressing noise and reverberation in the speech to be enhanced, and significantly improving the naturalness and clarity of the speech to be enhanced. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart illustrating a speech enhancement method provided in an embodiment of this application;

[0062] Figure 2A A schematic diagram of a speech enhancement model provided in an embodiment of this application;

[0063] Figure 2B A flowchart illustrating the training process of a speech enhancement model provided in this application embodiment;

[0064] Figure 3 This is a schematic diagram of a voice enhancement device provided in an embodiment of this application. Detailed Implementation

[0065] As described earlier, speech enhancement is typically performed using speech enhancement models based on Deep Neural Networks (DNNs). Speech enhancement models are mainly divided into two categories: generative models and discriminative models. Generative models learn the probability distribution of clean speech and reconstruct noisy speech into a speech signal consistent with that probability distribution. However, generative models suffer from mode collapse during training, meaning they may focus too much on certain specific speech features while ignoring others, resulting in a lack of naturalness in the generated speech signal. Discriminative models, on the other hand, suppress noise by minimizing the difference between the training speech output by the speech enhancement model and the corresponding clean speech after inputting noisy speech. However, discriminative models rely on a large amount of "noisy speech - clean speech" data pairs for training. When faced with unseen noise types, the discriminative model lacks the ability to learn these unseen noise types, making it difficult to effectively improve the clarity of the enhanced speech.

[0066] Through research, the inventors have proposed a speech enhancement method, apparatus, and device. This application learns a robust representation of the speech to be enhanced (i.e., acquires a noisy context representation) through a self-supervised module, and converts the noisy context representation into enhanced speech through the high-fidelity reconstruction capability of the vocoder module. This results in speech with higher naturalness and clarity (clean speech), effectively solving the speech distortion problem caused by pattern collapse in traditional generative models and the problem of discriminative models relying on large amounts of labeled data and having poor generalization. Furthermore, the self-supervised module learns noise-resistant representations from unlabeled data using contrastive loss and consistency loss, significantly improving the speech enhancement model's adaptability to unseen noise; the vocoder module ensures the naturalness and sound quality of the enhanced speech through multi-scale adversarial training and spectral constraints. Therefore, the speech enhancement method provided in this application can still generate high-clarity and high-naturalness speech in complex noisy environments, and does not rely on paired data, significantly reducing training costs.

[0067] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0068] See Figure 1 The figure is a flowchart of a speech enhancement method provided in an embodiment of this application. The method includes:

[0069] S101: Obtain the speech to be enhanced, which is noisy speech.

[0070] Noisy speech refers to raw speech that includes background noise, reverberation, or other interference, and needs to be enhanced through subsequent steps to restore clarity and naturalness.

[0071] S102: By inputting the speech to be enhanced into the self-supervised module of the speech enhancement model, the speech to be enhanced is converted into a noisy contextual representation.

[0072] The self-supervised module consists of a feature encoder and a transformer module (contextual encoder). Specifically, the self-supervised module inputs the speech to be enhanced into the speech enhancement model. First, the feature encoder extracts features from the speech to be enhanced to obtain noisy features. Then, the transformer module randomly masks the noisy features and obtains the noisy context representation (i.e., noise robustness features) based on the masked noisy features.

[0073] As is understandable, the above explains how, in practical applications, the speech to be enhanced is input into the self-supervised module of the speech enhancement model, converting the speech into a noisy contextual representation. Next, the training method of the self-supervised module will be explained:

[0074] See Figure 2A This figure is a schematic diagram of a speech enhancement model provided in an embodiment of this application. See also... Figure 2B The figure is a flowchart of the training process of a speech enhancement model provided in an embodiment of this application.

[0075] Depend on Figure 2A As can be seen, the speech enhancement model consists of a self-supervised module and a vocoder module connected in series. The self-supervised module adopts a two-branch structure (processing noisy speech samples and clean speech samples respectively), which is used to convert the speech waveforms of noisy speech samples and clean speech samples into corresponding noisy context representations and clean context representations, respectively.

[0076] For example, clean speech samples can come from datasets such as Librispeech, LibriVox, LibriTTS, and VCTK. Noisy speech samples can come from datasets such as Audioset, FreeSound, and FSD50K. This application does not limit the specific data source.

[0077] Specifically, the feature encoder consists of an M-layer one-dimensional convolutional neural network (e.g., 7 layers), with the stride and kernel size of each layer gradually decreasing (e.g., stride 5 and kernel size 10 for the first layer; stride 2 and kernel size 2 for the last layer). The feature encoder processes noisy speech samples x. noisy and clean speech samples x clean Feature extraction is performed separately to obtain the noisy features z of the sample. noisy and clean features of the sample z clean , i.e. z noisy =f(x) noisy ), z clean =f(x) clean The Transformer module consists of N Transformer blocks (e.g., 12 or 24 layers). The Transformer module handles noisy features z in samples. noisy and clean features of the sample z clean Random masks are applied separately (to force the speech enhancement model to learn noise robustness), and then each mask is input into a Transformer block to obtain the noisy context representation c of the samples. noisy and sample clean context representation c clean That is, c noisy =g(z) noisy ), c clean =g(z) clean ).

[0078] Subsequently, based on the noisy context representation c of the sample noisy Clean context representation of the sample c clean Noisy features of the sample z noisy Cleanliness feature of the sample z clean and the total loss function L total The self-supervised module is trained. The total loss function L of the self-supervised module is... total By contrastive loss function L con Consistency loss function L consis Composition. In a specific implementation, the total loss function can be represented by the following formula (1):

[0079] L total =L con +α1·L consis (1)

[0080] Wherein, α1 is the first proportional value.

[0081] Specifically, the comparison loss function L con The formula can be shown in formula (2) below:

[0082]

[0083] Where sim is the cosine similarity, t is the time index, k is the temperature parameter, and c noisy For the noisy context representation of the sample, z clean For clean features of the sample, z noisy This represents the noisy features of the sample. It is understandable that the contrastive loss function L... con Its function is to: pull in the noisy context representation of the sample c noisy and clean features of the sample z clean The similarity, and extrapolate the noisy context representation of the sample c noisy and sample noisy features z noisy The similarity enables the speech enhancement model to learn to extract features from noisy speech samples that are consistent with clean speech samples.

[0084] Consistency loss function L consis The formula can be shown in formula (3) below;

[0085] L consis =‖c noisy -c clean ||2 (3)

[0086] Among them, c noisy For the noisy context representation of the sample, c clean This represents a clean context for the sample. Understandably, the consistency loss function L... consis Its function is to constrain the consistency of the distribution of noisy and clean speech samples in the representation space, thereby improving the denoising effect.

[0087] S103: By inputting the noisy context representation into the vocoder module of the speech enhancement model, the noisy context representation is converted into enhanced speech, and the enhanced speech is clean speech.

[0088] The vocoder module includes a generator, which comprises a multi-layer one-dimensional transposed convolution and a multi-resolution fusion (MRF) module. Specifically, the vocoder module, which inputs the noisy context representation into the speech enhancement model, first upsamples the noisy context representation using multi-layer one-dimensional transposed convolution to obtain intermediate features at multiple time scales. The temporal resolution of each intermediate feature is aligned with the temporal resolution of the speech to be enhanced. Subsequently, the multi-resolution fusion module fuses the intermediate features from multiple time scales to obtain the enhanced speech.

[0089] The vocoder module also includes a discriminator, comprising a multi-scale discriminator (MSD) and a multi-period discriminator (MPD). After the multi-resolution fusion module fuses intermediate features from multiple time scales (resulting in fused speech), the multi-scale discriminator performs level evaluation on the fused speech at multiple resolution levels, and the multi-period discriminator performs component evaluation on the periodic components of the fused speech. When both the level evaluation and component evaluation meet their respective evaluation criteria, the fused speech is determined to be enhanced speech.

[0090] As is understandable, the above describes how, in practical applications, the noisy context representation is input into the vocoder module of the speech enhancement model to convert it into an interpretation of the enhanced speech. Next, the training method of the vocoder module will be explained:

[0091] Depend on Figure 2A It can be seen that the vocoder module includes a generator and two discriminators, used to represent the noisy context of the samples output by the self-supervised module as c. noisy Reconstruction is performed to obtain sample-enhanced speech.

[0092] Specifically, the generator includes multiple layers of one-dimensional transposed convolutions, which are used to represent the noisy context of the samples. noisy Upsampling k u The upsampling factor is increased progressively with each layer to make the sample noisy context representation c noisy The temporal resolution is aligned with that of the original noisy sample speech. Understandably, the output of the multi-layer one-dimensional transposed convolution represents the intermediate features of the sample after initial upsampling (before a waveform is formed). The generator also includes a multi-resolution fusion module, which fuses the intermediate features of the sample at different time scales to output the enhanced sample speech.

[0093] The discriminator includes a multi-scale discriminator and a multi-cycle discriminator. The input to the discriminator is a clean speech sample x. clean and sample-enhanced speech The discriminator outputs a probability value (0-1) representing the sample-enhanced speech. It is a clean speech sample x clean The confidence level. Among them, the multi-scale discriminator is used to enhance the speech of the samples. Perform multi-level downsampling (e.g., 2×, 4×, 8×) to evaluate the sample-enhanced speech from multiple resolution levels. To ensure authenticity and prevent the generator from producing speech that is "partially plausible but overall disjointed" (such as intermittent waveforms), a multi-period discriminator is used to enhance the speech samples. Segmentation (e.g., cycle lengths = 2, 3, 5, 7) is performed to evaluate the enhanced speech samples at multiple cycle levels. To ensure the authenticity of speech and prevent the generator from producing speech with "reasonable spectrum but chaotic phase" (such as mechanical sounds or phase distortion), it can be seen that the multi-scale discriminator and the multi-period discriminator constrain the generator from the two dimensions of "time domain multi-scale" and "frequency domain periodicity" respectively, thus covering all the physical characteristics of speech.

[0094] In one specific implementation, the generator's loss function L G By the first adversarial loss function L adv (G;D), Feature matching loss function L fm Mel spectrum loss function L mel Amplitude loss function L mag and complex loss function L com Composition. The generator's loss function L G It can be represented by the following formula (4):

[0095] L G =L adv (G;D)+β1L fm +β2L mel +β3L mag +β4L com (4)

[0096] Wherein, β1, β2, β3 and β4 are the first hyperparameter, the second hyperparameter, the third hyperparameter and the fourth hyperparameter, respectively.

[0097] Specifically, the first adversarial loss function is used to address the oversmoothing problem and improve the sample-enhanced speech. Global authenticity, preserving samples to enhance speech High-frequency details (such as plosive sounds). First adversarial loss function L adv (G;D) can be represented by the following formula (5):

[0098] L adv (G;D)=E[(D(G(c noisy ))-1) 2 (5)

[0099] Feature matching loss function L fm Used to constrain clean speech samples x clean and sample-enhanced speech The distance in the discriminator's feature space (i.e., minimizing feature differences) prevents the generator from optimizing only the adversarial loss while ignoring the rationality of the speech structure (preventing pattern collapse). The feature matching loss function L... fm It can be represented by the following formula (6):

[0100]

[0101] Where T represents the number of discriminator layers, D i and N i These represent the features and the number of features in the i-th layer of the discriminator, respectively.

[0102] Mel spectrum loss function L mel Used to minimize clean speech samples x clean and sample-enhanced speech The L1 distance in the Mel frequency domain ensures sample-enhanced speech. Auditory perception quality (such as timbre and intelligibility) and clean speech samples x clean Consistent. Mel spectral loss function L mel It can be represented by the following formula (7):

[0103]

[0104] in, This is the mapping function from speech signals to the Mel spectrum.

[0105] Amplitude loss function L mag Used for minimizing clean speech samples x clean and sample-enhanced speech The difference in amplitude spectrum from the short-time Fourier transform (STFT) is used to constrain the frequency domain energy distribution and avoid spectral distortion. Amplitude loss function L mag It can be represented by the following formula (8):

[0106]

[0107] Where STFT(·) is the amplitude spectrum of the short-time Fourier transform.

[0108] Complex loss function L com This method is used to simultaneously minimize the difference between the real (amplitude) and imaginary (phase) parts of the STFT, solving the problem of inaccurate phase estimation in traditional methods and improving waveform reconstruction accuracy. Complex loss function L com It can be represented by the following formula (9):

[0109]

[0110] Where Re(·) and Im(·) are the real and imaginary parts (i.e., phase information) of the STFT, respectively.

[0111] In one specific implementation, the discriminant's loss function L D The second adversarial loss function can be represented by the following formula (10):

[0112]

[0113] Therefore, after obtaining the clean speech samples and the enhanced speech samples, the generator of the vocoder module can be trained based on the clean speech samples, the enhanced speech samples, and the generator's loss function; and the discriminator of the vocoder module can be trained based on the clean speech samples, the enhanced speech samples, and the second adversarial loss function.

[0114] In summary, this application discloses a speech enhancement method. This method learns a robust representation of the speech to be enhanced (i.e., acquires a noisy context representation) through a self-supervised module, and converts the noisy context representation into enhanced speech through the high-fidelity reconstruction capability of the vocoder module. This results in speech with higher naturalness and clarity (clean speech), effectively solving the speech distortion problem caused by pattern collapse in traditional generative models and the problems of discriminative models relying on large amounts of labeled data and having poor generalization. Furthermore, the self-supervised module learns noise-resistant representations from unlabeled data using contrastive loss and consistency loss, significantly improving the speech enhancement model's adaptability to unseen noise. The vocoder module ensures the naturalness and sound quality of the enhanced speech through multi-scale adversarial training and spectral constraints. Therefore, the speech enhancement method provided in this application can still generate high-clarity and highly natural speech in complex noisy environments, and does not rely on paired data, significantly reducing training costs.

[0115] See Figure 3 This figure is a schematic diagram of a voice enhancement device provided in an embodiment of this application. The voice enhancement device 300 includes: a voice acquisition module 301, a first conversion module 302, and a second conversion module 303;

[0116] The speech acquisition module 301 is used to acquire the speech to be enhanced, which is noisy speech.

[0117] The first conversion module 302 is used to convert the speech to be enhanced into a noisy contextual representation by inputting the speech to be enhanced into the self-supervised module of the speech enhancement model;

[0118] The second conversion module 303 is used to convert the noisy context representation into enhanced speech by inputting the noisy context representation into the vocoder module of the speech enhancement model, and the enhanced speech is clean speech.

[0119] Optionally, the self-supervised module includes a feature encoder and a Transformer module; the first transformation module includes 302: a first transformation submodule and a second transformation submodule.

[0120] The first conversion submodule is used to extract features from the speech to be enhanced using a feature encoder to obtain noisy features.

[0121] The second transformation submodule is used to randomly mask the noisy features through the Transformer module and obtain the noisy context representation based on the masked noisy features.

[0122] Optionally, the vocoder module includes a generator, which includes a multi-layer one-dimensional transposed convolution and a multi-resolution fusion module; the second conversion module 303 includes a third conversion submodule and a fourth conversion submodule;

[0123] The third transformation submodule is used to upsample the noisy context through multi-layer one-dimensional transposed convolution to obtain intermediate features at multiple time scales. The temporal resolution of each intermediate feature is aligned with the temporal resolution of the speech to be enhanced.

[0124] The fourth conversion submodule is used to fuse intermediate features from multiple time scales through a multi-resolution fusion module to obtain enhanced speech.

[0125] Optionally, the vocoder module also includes a discriminator, which includes a multi-scale discriminator and a multi-period discriminator; the fourth conversion submodule is specifically used for:

[0126] By using a multi-resolution fusion module, intermediate features from multiple time scales are fused to obtain fused speech.

[0127] The fused speech is evaluated at multiple resolution levels using a multi-scale discriminator, and the periodic components of the fused speech are evaluated using a multi-period discriminator.

[0128] When both the level assessment and the component assessment meet the corresponding assessment criteria, the fused speech is determined to be enhanced speech.

[0129] Optionally, the training units of the self-supervised module are as follows:

[0130] The first training unit is used to acquire noisy speech samples and clean speech samples.

[0131] The second training unit is used to extract features from noisy speech samples and clean speech samples respectively through the feature encoder, and obtain noisy features and clean features of the samples.

[0132] The third training unit is used to randomly mask the noisy features and clean features of the samples through the Transformer module, and to obtain the noisy context representation and the clean context representation of the samples based on the masked noisy features and the masked clean features, respectively.

[0133] The fourth training unit is used to train the self-supervised module based on the noisy context representation of the sample, the clean context representation of the sample, the noisy features of the sample, the clean features of the sample, and the total loss function. The total loss function includes the contrastive loss function and the consistency loss function.

[0134] Optionally, the training unit of the vocoder module is as follows:

[0135] The fifth training unit is used to upsample the noisy context of the sample through multi-layer one-dimensional transposed convolution to obtain intermediate features of the sample at multiple time scales. The temporal resolution of each intermediate feature is aligned with the temporal resolution of the noisy speech sample.

[0136] The sixth training unit is used to fuse intermediate features of samples from multiple time scales through the multi-resolution fusion module to obtain sample-enhanced speech.

[0137] The seventh training unit is used to train the generator of the vocoder module based on the clean speech samples, the enhanced speech samples, and the generator's loss function. The generator's loss function includes the first adversarial loss function, the feature matching loss function, the Mel spectrum loss function, the amplitude loss function, and the complex loss function.

[0138] The eighth training unit is used to train the discriminator of the vocoder module based on clean speech samples, enhanced speech samples, and the second adversarial loss function.

[0139] In summary, this application discloses a speech enhancement device. This application learns a robust representation of the speech to be enhanced (i.e., acquires a noisy context representation) through a self-supervised module, and converts the noisy context representation into enhanced speech through the high-fidelity reconstruction capability of the vocoder module. This results in speech with higher naturalness and clarity (clean speech), effectively solving the speech distortion problem caused by pattern collapse in traditional generative models, and the problem of discriminative models relying on large amounts of labeled data and having poor generalization. Furthermore, the self-supervised module learns noise-resistant representations from unlabeled data using contrastive loss and consistency loss, significantly improving the speech enhancement model's adaptability to unseen noise; the vocoder module ensures the naturalness and sound quality of the enhanced speech through multi-scale adversarial training and spectral constraints. Therefore, the speech enhancement device provided in this application can still generate high-clarity and highly natural speech in complex noisy environments, and does not rely on paired data, significantly reducing training costs.

[0140] This application also provides a corresponding speech enhancement device for implementing the speech enhancement method provided in this application.

[0141] The voice enhancement device includes a memory and a processor. The memory is used to store instructions or code, and the processor is used to execute the instructions or code to cause the device to perform a voice enhancement method according to any embodiment of this application.

[0142] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech enhancement method, characterized in that, The method includes: Acquire the speech to be enhanced, wherein the speech to be enhanced is noisy speech; By inputting the speech to be enhanced into the self-supervised module of the speech enhancement model, the speech to be enhanced is converted into a noisy contextual representation; By inputting the noisy context representation into the vocoder module of the speech enhancement model, the noisy context representation is converted into enhanced speech, which is clean speech; The self-supervised module includes a feature encoder and a Transformer module; the conversion of the speech to be enhanced into a noisy context representation includes: The feature encoder is used to extract features from the speech to be enhanced to obtain noisy features; The noisy features are randomly masked using the Transformer module, and a noisy context representation is obtained based on the masked noisy features. The vocoder module includes a generator, which includes a multi-layer one-dimensional transposed convolution and a multi-resolution fusion module; the conversion of the noisy context representation into enhanced speech includes: The noisy context is upsampled by the multi-layer one-dimensional transposed convolution to obtain intermediate features at multiple time scales, and the temporal resolution of each intermediate feature is aligned with the temporal resolution of the speech to be enhanced. The multi-resolution fusion module fuses intermediate features from multiple time scales to obtain enhanced speech.

2. The method according to claim 1, characterized in that, The vocoder module further includes a discriminator, which includes a multi-scale discriminator and a multi-period discriminator; the step of fusing intermediate features from the multiple time scales through the multi-resolution fusion module to obtain enhanced speech includes: The multi-resolution fusion module fuses intermediate features from multiple time scales to obtain fused speech. The multi-scale discriminator performs level evaluation of the fused speech at multiple resolution levels, and the multi-period discriminator performs component evaluation of the periodic components of the fused speech. When both the level assessment and the component assessment meet the corresponding assessment conditions, the fused speech is determined to be enhanced speech.

3. The method according to claim 2, characterized in that, The training method for the self-supervised module is as follows: Acquire noisy and clean speech samples; The feature encoder is used to extract features from the noisy speech sample and the clean speech sample, respectively, to obtain noisy speech sample features and clean speech sample features. The Transformer module randomly masks the noisy features and clean features of the sample, and obtains the noisy context representation and clean context representation of the sample based on the masked noisy features and clean features, respectively. The self-supervised module is trained based on the noisy context representation of the sample, the clean context representation of the sample, the noisy features of the sample, the clean features of the sample, and the total loss function, wherein the total loss function includes a contrastive loss function and a consistency loss function.

4. The method according to claim 3, characterized in that, The training method for the vocoder module is as follows: The sample noisy context is upsampled by the multi-layer one-dimensional transposed convolution to obtain sample intermediate features at multiple time scales. The temporal resolution of each sample intermediate feature is aligned with the temporal resolution of the sample noisy speech. The multi-resolution fusion module fuses intermediate features of samples from multiple time scales to obtain sample-enhanced speech. The generator of the vocoder module is trained based on the clean speech sample, the enhanced speech sample, and the generator's loss function. The generator's loss function includes a first adversarial loss function, a feature matching loss function, a Mel-spectrum loss function, an amplitude loss function, and a complex loss function. The discriminator of the vocoder module is trained based on the clean speech samples, the enhanced speech samples, and the second adversarial loss function.

5. A voice enhancement device, characterized in that, The device includes: a voice acquisition module, a first conversion module, and a second conversion module; The speech acquisition module is used to acquire the speech to be enhanced, which is noisy speech; The first conversion module is used to convert the speech to be enhanced into a noisy contextual representation by inputting the speech to be enhanced into a self-supervised module of the speech enhancement model; The second conversion module is used to convert the noisy context representation into enhanced speech by inputting the noisy context representation into the vocoder module of the speech enhancement model, wherein the enhanced speech is clean speech; The self-supervised module includes a feature encoder and a Transformer module; the first transformation module includes: a first transformation submodule and a second transformation submodule. The first conversion submodule is used to extract features from the speech to be enhanced through the feature encoder to obtain noisy features; The second transformation submodule is used to randomly mask the noisy features through the Transformer module, and obtain a noisy context representation based on the masked noisy features; The vocoder module includes a generator, which includes a multi-layer one-dimensional transposed convolution and a multi-resolution fusion module; the second conversion module includes a third conversion submodule and a fourth conversion submodule. The third transformation submodule is used to upsample the noisy context through the multi-layer one-dimensional transposed convolution to obtain intermediate features at multiple time scales, and the temporal resolution of each intermediate feature is aligned with the temporal resolution of the speech to be enhanced. The fourth conversion submodule is used to fuse intermediate features from multiple time scales through the multi-resolution fusion module to obtain enhanced speech.

6. A voice enhancement device, characterized in that, The device includes: a memory and a processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the speech enhancement method as described in any one of claims 1 to 4.

Images