Noise robust speech synthesis method based on diffusion auto-encoder

Abstract

The invention belongs to the field of speech synthesis, and relates to a noise robust speech synthesis method based on a diffusion auto-encoder. Comprising the following steps: constructing a noise data set, and extracting Mel spectrum features from noise audios and clean audios as a training set and a test set; performing diffusion auto-encoder training, and training a diffusion auto-encoder model by using noisy Mel spectrum features as input and clean Mel spectrum features as output so as to generate noise robust hidden features; performing text-to-feature model training, training a model by using a text data set and hidden features extracted by a diffusion auto-encoder, the model being obtained by further processing based on a FastSpeech2 model; reasoning a speech synthesis model, inputting a test text, generating features from the trained text to a feature model, generating a Mel spectrum through a conditional diffusion model, and finally restoring the Mel spectrum into a speech waveform by using a vocoder. According to the method, the low-quality noisy data is utilized to synthesize the voice with high naturalness and high quality.

Description

Technical Field

[0001] The invention belongs to the field of speech synthesis and provides a noise-robust speech synthesis method. A diffusion autoencoder is trained based on a low-quality noisy data set to extract noise-independent features for synthesizing clean speech. Background Art

[0002] End-to-end speech synthesis trained on publicly available high-quality datasets is able to generate high-quality speech in a variety of application scenarios. However, in everyday scenarios, such as in smart home environments, the collected speech data exhibits relatively low quality. Although these data do not have the same quality as speech recorded in professional studios, they still contain rich semantics and are very beneficial for the development of domain-specific speech interaction systems.

[0003] For methods that use low-quality data to train robust speech synthesis, it is common to enhance the audio quality of the dataset through speech enhancement techniques. For example, a speech enhancement model enhances low-quality noisy data to enhance the training dataset of the speech synthesis model and serves as a preprocessing step for speech synthesis. However, most speech enhancement models are not always well suited for speech synthesis tasks. When enhancing speech, it may cause additional speech distortion and impair speaker characteristics, especially in low signal-to-noise ratio environments, resulting in degraded speech quality. In addition to directly adopting speech enhancement methods, self-supervised learning of speech representations using large models has become a promising solution for robust speech synthesis tasks. Self-supervised models are trained using large amounts of data, which may contain rich semantic information. Some methods use self-supervised model representations as a substitute for traditional acoustic features to achieve robust speech synthesis. Although self-supervised learning provides a potential solution to the challenges in robust speech synthesis, it still suffers from the following shortcomings: First, there is a domain adaptation problem. The differences between the self-supervised pre-training corpus and the speech synthesis training data, such as language variation, may affect the quality of speech synthesis; in addition, although self-supervised representations contain rich semantic information, they may cause leakage of the source speaker, potentially affecting the speaker similarity in the synthesized speech.

[0004] This method uses low-quality noisy data to train a diffusion autoencoder model to learn noise-robust features and utilizes the features for speech synthesis. Summary of the invention

[0005] The technical problem solved by the present invention is to provide a robust speech synthesis technology for low-quality noise data. The method adopts a diffusion autoencoder and a text-to-feature model, makes full use of low-quality speech data, and can synthesize highly natural and high-quality speech.

[0006] To address the deficiencies mentioned in the above background technology, the technical solution of the present invention is: a noise robust speech synthesis method based on a diffusion autoencoder, comprising the following four steps: 1) constructing a noise data set, extracting Mel spectrum features from noisy audio and clean audio as a training set and a test set; 2) using noisy Mel spectrum features as input and clean Mel spectrum features as output to train a diffusion autoencoder model; 3) using a text data set as input and features as target output to train a text-to-feature model; 4) performing inference on a speech synthesis model, inputting text, generating robust features through a text-to-feature model, generating Mel spectrum based on the features through a conditional diffusion model, and finally generating a speech waveform through a vocoder using the Mel spectrum.

[0007] The specific steps are as follows:

[0008] First: Build a noisy dataset

[0009] Select the LJSpeech dataset as the clean dataset and the Freesound dataset as the source of noise data. Add the noise data to the LJSpeech dataset with a signal-to-noise ratio of 5 dB to 20 dB to construct a noisy dataset. Mix the clean dataset and the noisy dataset as a mixed dataset. Convert all audio sampling rates to 16 kHz. Extract Mel-spectrogram features from all audio data in the training set and test set as input and target output data for training the model.

[0010] Second: Training the Diffusion Autoencoder Model

[0011] The diffusion autoencoder model consists of a feature extractor and a conditional diffusion model. It uses a mixed dataset consisting of clean Mel-spectrogram features and noisy Mel-spectrogram features as input data, and the corresponding clean Mel-spectrogram features as target output data.

[0012] During the whole diffusion autoencoder training process, the feature extractor generates latent features based on the input Mel spectrum features;

[0013] According to the extracted latent features, the conditional diffusion model adds noise to the target Mel spectrum features during training to generate low-dimensional noise information, thereby learning the distribution of noise;

[0014] During the inference process, the feature extractor generates noise-robust latent features based on the input noisy Mel spectrum features. Then the conditional diffusion model generates low-dimensional random noise, and then decodes the random noise to generate clean Mel spectrum features using the latent features as conditions.

[0015] By using mixed Mel-spectrogram features as input and clean Mel-spectrogram features as output to simultaneously train the feature extractor and conditional diffusion model, the diffusion autoencoder is able to generate noise-robust latent features.

[0016] The trained feature extractor can generate noise-robust latent features from low-quality data, and these latent features can be restored to clean Mel-spectrogram features through the conditional diffusion model;

[0017] Finally, the HiFi-GAN vocoder is used to restore the Mel-spectrogram features to speech waveform.

[0018] Third: Training text to feature model

[0019] The text-to-feature model is modified from the FastSpeech2 model, which includes a phoneme embedding module, an encoder, a variance adapter, and a feature decoder.

[0020] During the training process, text is used as input data, and the features extracted by the feature extractor in the diffuse encoder are used as target output data;

[0021] First, the text-to-feature model converts text into phonemes, and then uses the phoneme embedding module to convert phonemes into phoneme sequences;

[0022] The encoder converts the input phoneme sequence into corresponding latent features;

[0023] Then, the variance adapter adds duration, fundamental frequency, and energy information to the phoneme latent sequence to ensure that the target features can be synthesized;

[0024] Finally, the feature decoder transforms the latent sequence into robust features.

[0025] Fourth: Noise-Robust Speech Synthesis Model

[0026] Perform inference on the speech synthesis model, input the test text, pass the trained text to the feature model, generate features, then pass it through the conditional diffusion model to generate the Mel spectrum, and finally use the vocoder to restore the Mel spectrum to the speech waveform.

[0027] Beneficial effects:

[0028] The present invention uses low-quality data to train a diffusion autoencoder model to learn robust features.

[0029] The method proposed in the present invention can utilize low-quality data sets to generate noise-robust features, thereby synthesizing high-quality and highly natural speech.

[0030] The present invention designs a text-to-feature prediction module for an end-to-end speech synthesis method, which can convert text into features. The method simplifies the speech synthesis task into two parts: mapping text to noise-robust features and mapping features to speech. In the process of speech synthesis model inference, text is input, and noise-robust features are generated through the text-to-feature prediction module. The features are then passed through a conditional diffusion model to generate mel-spectrogram features. Finally, a vocoder is used to restore the mel-spectrogram features to speech waveforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to make the description of the implementation mode of the present invention clearer, the following will introduce the legends needed to describe the examples of the present invention.

[0032] Figure 1 Flowchart for training the diffusion autoencoder model;

[0033] Figure 2 Flowchart of the end-to-end speech synthesis model. DETAILED DESCRIPTION

[0034] The following is a demonstration of the functions and effects of the present invention in conjunction with the accompanying drawings and tables. The specific steps are as follows:

[0035] 1) Construct a noisy dataset

[0036] The single-speaker dataset LJSpeech is selected as the training dataset, which contains 13,100 data items. In order to simulate the real noise environment, noise data is added to the LJSpeech dataset, and the noise dataset is simulated with a signal-to-noise ratio of 5 dB to 20 dB, where the noise dataset comes from the Freesound dataset. All audio sampling rates are converted to 16 kHz. The clean LJSpeech dataset is divided into 12,822 items as a clean training set and 278 items as a clean test set. Similarly, the noisy LJSpeech dataset is divided into 12,822 items as a noisy training set and 278 items as a noisy test set. The clean training set and the noisy training set are mixed as a mixed training set. During the training and testing process, the speech data needs to be converted into Mel-spectrogram features.

[0037] 2) Training the diffusion autoencoder model

[0038] The diffusion autoencoder model consists of a feature extractor and a text-to-feature prediction module; the goal of the feature extractor is to extract more abstract deep features from the mel-spectrogram for training and learning the conditional diffusion model; the two-dimensional convolution in the feature extractor is changed to a one-dimensional convolution, and the up and down sampling is removed from the feature extractor to model the time dependency; the feature extractor consists of a ResBlock network, an AttentionBlock network, a GroupNorm32 network, a SiLU network, and a one-dimensional convolutional network, such as Figure 1 As shown in the figure, the ResBlock network consists of a GroupNorm32 network, a SiLU network, and a one-dimensional convolutional network, and the AttentionBlock network consists of a GroupNorm32 network, a one-dimensional convolutional network layer, a QKVAttentionLegacy network, and a one-dimensional convolutional network. The ResBlock network and the AttentionBlock network can extract necessary features other than noise from low-quality data and model temporal dependencies. The feature extractor extracts noise-robust features with 512 dimensions.

[0039] The conditional diffusion model is built based on the UNet network. During the training process, the conditional diffusion encoder uses the latent feature h feas As a condition, the target Mel spectrum feature is encoded into a low-dimensional feature x T , which contains information that the latent feature does not have; during the training process, the loss function L of θ is optimized simple as follows:

[0040]

[0041] in α t is calculated by the noise level, θ represents the UNet network, T represents the number of steps of conditional diffusion model training, and x t It represents the information obtained by adding noise to the target Mel spectrum at t steps during the training process.

[0042] In the inference process, the present invention uses the conditional diffusion model to extract the latent features h according to the feature extractor. feas and the generated random noise x T Predict the target Mel spectrum; the conditional diffusion model is constructed by modeling p θ (x t-1 |x t ,h feas ) Based on the latent features, the clean Mel spectrum feature X0 is predicted as follows:

[0043]

[0044] During the entire diffusion autoencoder training process, the feature extractor generates latent features based on the input Mel spectrum features. Based on the extracted latent features, the conditional diffusion model adds noise to the target Mel spectrum features during the training process to generate low-dimensional noise information, thereby learning the distribution of noise.

[0045] During the inference process, the feature encoder generates noise-robust latent features based on the input noisy Mel-spectrogram features. Then the conditional diffusion model generates low-dimensional random noise, and then decodes the random noise to generate clean Mel-spectrogram features using the latent features as conditions.

[0046] By using noisy Mel-spectrogram features as input and clean Mel-spectrogram features as output to simultaneously train the feature extractor and the conditional diffusion model, the diffusion autoencoder can generate noise-robust latent features; the trained feature extractor can extract noise-robust latent features from low-quality data, and these latent features can be restored to clean Mel-spectrogram features through the conditional diffusion model; finally, the HiFi-GAN vocoder is used to restore the Mel-spectrogram features to speech waveforms.

[0047] 3) Training text-to-feature model

[0048] The text-to-feature model is modified based on the FastSpeech2 model, which includes a phoneme embedding module, an encoder, a variance adapter, and a feature decoder;

[0049] During the training process, text is used as input data, and the features extracted by the feature extractor in the diffuse encoder are used as target output data to train the text-to-feature model.

[0050] First, the input text is converted into phonemes, and then the phonemes are converted into phoneme sequences using the phoneme embedding module;

[0051] The encoder converts the input phoneme sequence into corresponding features;

[0052] Then, the variance adapter adds duration, fundamental frequency and energy information to the phoneme latent sequence. The variance adapter consists of a duration predictor, a fundamental frequency predictor and an energy predictor to ensure that the target features can be synthesized.

[0053] Finally, the feature decoder converts the latent sequence into features, which are used as input by the conditional diffusion model to generate clean Mel-spectrogram features, which are finally converted into speech waveforms using the HiFi-GAN vocoder;

[0054] By training a text-to-feature model using paired text and noise-robust features, the model is able to generate features suitable for a conditional diffusion model based on the text, thereby generating clean mel-spectrogram features that are then converted into speech waveforms.

[0055] 4) Noise-robust speech synthesis model

[0056] The method of the present invention simplifies the speech synthesis task into two parts: mapping of text to noise-robust features and mapping of features to speech; in the process of speech synthesis model inference, text is input, and noise-robust features are generated through a text-to-feature prediction module, and the features are used to generate Mel-spectrogram features through a conditional diffusion model; finally, a vocoder is used to restore the Mel-spectrogram features to a speech waveform.

[0057] The present invention uses low-quality data to train a diffusion autoencoder model to learn robust features; the present invention uses the following parameters to configure the diffusion autoencoder: the number of time embedding channels and style channels is set to 512, the attention head is set to 1, the number of repeated ResBlocks of each resolution is set to 2, and T is set to 1000 during training; the diffusion autoencoder is trained for a total of 300,000 steps with a batch size of 16; the text-to-feature prediction module is trained for a total of 900,000 steps with a batch size of 16; the HiFi-GAN vocoder is trained for a total of 800,000 steps with a batch size of 16.

[0058] In order to evaluate the speech generated by the method proposed in the present invention, the present invention selects the mean opinion score (MOS) as a subjective indicator, and selects background noise (BN), background noise average score (DNSMOS), naturalness (NISQA) and speaker similarity as objective indicators; MOS is used to evaluate the overall quality of the audio, and the MOS value with a 95% confidence interval is calculated according to the t-distribution. The higher the MOS value, the better the quality of the synthesized speech; the higher the BN value, the lower the background noise; the DNSMOS value represents a comprehensive indicator of audio quality and background noise; the present invention uses the NISQA tool to evaluate the naturalness of the speech, and the higher the value, the higher the naturalness of the speech.

[0059] The speech evaluation results enhanced by the diffusion autoencoder are shown in Table 1. Regardless of whether a clean or noisy test set is used, the diffusion autoencoder model trained with the mixed dataset performs better than other models, with MOS values ​​of 4.40±0.051 and 4.39±0.053, respectively, exceeding the MOS values ​​of 4.37±0.058 and 4.27±0.068 of the baseline model trained with the same amount of data, proving that the method proposed in the present invention is stronger than the baseline model; among the three proposed models, the model trained with the mixed dataset exhibits the lowest background noise, and The BN score of the proposed method is 4.56, which exceeds the model trained with only noise data (4.28) and the baseline model (4.12). In addition, 4.56 is close to the BN value of 4.65 for real clean audio and significantly higher than the BN value of 3.51 for real noise audio. It is worth noting that when the test set is a noise data set, the DNSMOS value synthesized by the model trained with the mixed data set is 4.19, which is better than the baseline model trained with the same amount of data. The speaker similarity of the model trained with the clean data set, the noise data set and the mixed data set respectively exceeds the baseline model. Therefore, the method of training the diffusion autoencoder proposed in the present invention can generate noise-robust features using low-quality noisy data, thereby achieving the effect of noise reduction while retaining the naturalness and speaker similarity of the original speech.

[0060] Table 1 Speech evaluation results of the diffusion autoencoder model and the baseline model enhancement

[0061]

[0062] The present invention connects the text-to-feature model and the conditional diffusion model, first converts the text into features, then converts the features into mel-spectrograms, and then uses the HiFi-GAN vocoder to convert the mel-spectrograms into speech waveforms, and evaluates the synthesized speech. The experimental results are shown in Table 2.

[0063] Table 2 Performance of models trained with different datasets in end-to-end speech synthesis tasks

[0064]

[0065] It is obvious that the diffusion autoencoder model trained with the mixed dataset performs best in the end-to-end speech synthesis task test. The model is superior to the diffusion autoencoder model and the baseline model trained with the noisy dataset in terms of MOS value, background noise, DNSMOS value and speech naturalness. The experimental results show that the method proposed in the present invention can use low-quality datasets to generate noise-robust features, thereby synthesizing high-quality and high-naturalness speech.

Claims

1. A noise-robust speech synthesis method based on a diffusion autoencoder, characterized in that: The steps include: S1. Build a noise dataset and extract Mel-spectrogram features from noisy audio and clean audio as training and test sets. S2, using the noisy Mel spectrum features as input and the clean Mel spectrum features as target output to train the diffusion autoencoder model; S3, use the text dataset as input and the robust features as output to train the text-to-feature model; S4: Perform inference on the speech synthesis model, input text, generate features through the text-to-feature model, generate Mel spectrum based on the features through the diffusion autoencoder, and finally generate a speech waveform through the Mel spectrum through the vocoder.

2. The method according to claim 1, characterized in that: In the step S1, the open source dataset LJSpeech is selected as a clean dataset, noise is added to the LJSpeech dataset, and the signal-to-noise ratio is -5 decibels to 20 decibels to simulate a real noise environment and construct a noisy dataset; for the clean speech dataset and the noisy speech dataset, Mel-spectrogram features are extracted as training sets and test sets.

3. The method according to claim 1, characterized in that: In the step S2, the noisy Mel spectrum features are used as input and the clean Mel spectrum features are used as output to train the diffusion autoencoder model; the diffusion autoencoder is composed of a feature extractor and a conditional diffusion model; the goal of the feature extractor is to extract more abstract deep features from the Mel spectrum for training and learning the conditional diffusion model, and the feature extractor is composed of a ResBlock network, an AttentionBlock network, a GroupNorm32 network, and a SiLU network, which can extract necessary features other than noise from low-quality data and model temporal dependencies; the conditional diffusion model is based on a UNet network, and uses the latent features extracted by the feature extractor as conditions to encode the clean Mel spectrum features into low-dimensional features, which contain information other than latent features, thereby learning the distribution of noise; During the inference process, the feature encoder generates noise-robust latent features based on the input noisy Mel-spectrogram features. Then the conditional diffusion model generates low-dimensional random noise, and then decodes the random noise to generate clean Mel-spectrogram features using the latent features as conditions. By using noisy Mel-spectrogram features as input and clean Mel-spectrogram features as output to train the feature extractor and conditional diffusion model simultaneously, the diffusion autoencoder can generate noise-robust latent features. The trained feature extractor can extract noise-robust latent features from low-quality data, and these latent features can be restored to clean Mel spectrum features through the conditional diffusion model; Finally, the HiFi-GAN vocoder is used to restore the Mel-spectrogram features to speech waveform.

4. The method according to claim 1, characterized in that: In step S3, the text-to-feature model is modified based on the FastSpeech2 model, and the model includes a phoneme embedding module, an encoder, a variance adapter, and a feature decoder; during the training process, the text is used as input data, and the features extracted by the feature extractor in the diffuse encoder are used as target output data; First, the text-to-feature model converts text into phonemes, and then uses the phoneme embedding module to convert phonemes into phoneme sequences; the encoder converts the input phoneme sequence into corresponding latent features; Then, the variance adapter adds duration, fundamental frequency, and energy information to the phoneme latent sequence to ensure that the target features can be synthesized; Finally, the feature decoder transforms the latent sequence into features.

5. The method according to claim 1, characterized in that: In the step S4, low-quality data is used to train the diffusion autoencoder model to learn robust features; a text-to-feature prediction module is designed for the end-to-end speech synthesis method, which can convert text into features; the speech synthesis task is simplified into two parts: mapping of text to noise-robust features and mapping of features to speech; during the speech synthesis model inference process, text is input, and noise-robust features are generated through the text-to-feature prediction module, and the features are generated into Mel-spectrogram features through the conditional diffusion model; Finally, a vocoder is used to restore the Mel-spectrogram feature to a speech waveform.

Images