Speech synthesis method based on self-supervised learning and mutual information decoupling technology

Abstract

The present invention discloses a speech synthesis method based on self-supervised learning and mutual information decoupling technology, which mainly includes data preprocessing; using HUBERT and wav2vec models as pre-trained large models, and adopting a large number of unlabeled alignment methods for training; designing two tasks, speaker classification and style classification, as downstream tasks, fixing the weight parameters of the training model to obtain task-related feature representations; using mutual information to classify T speaker and T style Decouple; learn the T speaker and T style This approach is added to an end-to-end speech synthesis model that uses an encoder-attention-decoder architecture. This improves the quality of multi-speaker and multi-style speech synthesis, while also enhancing the synthesis model's ability to quickly adapt to small amounts of data.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis, and relates to a speech synthesis method based on self-supervised learning and mutual information decoupling technology, and specifically to optimizing a multi-speaker and multi-style speech synthesis system using a self-supervised pre-training method. Background Art

[0002] Speech synthesis, also known as text-to-speech (TTS), is a technology that converts arbitrary input text into corresponding speech. Traditional speech synthesis systems typically consist of two modules: a front-end and a back-end. The front-end module primarily analyzes the input text and extracts linguistic information required by the back-end module. For Chinese synthesis systems, the front-end module typically includes submodules such as text regularization, word segmentation, part-of-speech prediction, polyphone disambiguation, and prosody prediction. Based on the front-end analysis results, the back-end module generates speech waveforms using specific methods. Back-end systems are generally categorized into speech synthesis based on statistical parameter modeling (also known as parametric synthesis) and speech synthesis based on unit selection and waveform splicing (also known as concatenative synthesis). With the emergence and increasing maturity of end-to-end technology, single-speaker speech synthesis systems have been able to synthesize speech with a naturalness similar to human pronunciation. However, in real-world applications, single-speaker speech synthesis systems are no longer able to meet people's daily needs.

[0003] For example, in e-book reading, it is often necessary to synthesize audio for different characters, emotions, or styles. Existing speech synthesis methods often require large datasets for training to synthesize high-quality audio of the target style and speaker. Furthermore, they cannot clearly separate different styles and characters. Especially when there is only a small amount of reference audio for fine-tuning, the quality of the synthesized audio cannot be guaranteed.

[0004] While deep neural network models have achieved breakthroughs in various speech tasks over the past few years, they remain constrained by the large amounts of labeled data required for model training. The emergence of self-supervised pre-training methods has alleviated this problem to some extent. This method first uses large-scale unsupervised data for pre-training, then fine-tunes the trained model on smaller amounts of labeled data. Previous studies have shown that using self-supervised pre-training can improve performance in various speech tasks. Summary of the Invention

[0005] To solve the technical problems raised in the background technology, the present invention proposes a speech synthesis method based on self-supervised learning and mutual information decoupling technology, which mainly designs the following four aspects:

[0006] 1) A large model is trained using self-supervised training to extract general speech representations, which are learned through a learnable attention mechanism module.

[0007] 2) Design two downstream tasks, speaker identification and style identification, to convert general speech representations into task-related speech representations.

[0008] 3) When training the above two downstream tasks, mutual information is added as a constraint to decouple speaker representation and style representation.

[0009] 4) Finally, speaker representation and style representation are applied to the end-to-end speech synthesis model to realize a multi-speaker and multi-style speech synthesis system.

[0010] Step 1: Data preprocessing: Text requires front-end processing, typically using characters as input. Text and audio pairs can be used as training data. Preprocessing also requires extraction of mel-spectrogram features.

[0011] For text preprocessing, Python's natural language processing toolkit NLTK is usually used to perform operations such as word segmentation and Chinese character pinyin conversion. For the extraction of Mel-spectrogram features, Python's common audio processing toolkit librosa can be directly used to extract Mel-spectrogram features from audio through processes such as framing, windowing, and pre-emphasis.

[0012] Step 2: Use self-supervised learning to train the large model. Models such as HUBERT and wav2vec are used as pre-trained large models, and a large number of unlabeled alignment methods are used for training.

[0013] In step three, we design speaker classification and style classification as downstream tasks, and fix the weight parameters of the model trained in step two to obtain task-related feature representations.

[0014] 1) The large model in step 2 contains an N-layer neural network, and the output of the N-layer network [P1, P2, ..., P N ] will be used to learn the final task-related feature representation;

[0015] 2) Design an attention mechanism module, which is a learnable neural network layer that is used to weight the output of the N-layer neural network to obtain a temporary general feature representation P F , as shown in the following formula:

[0016] P F =Atten(P1,P2,…,P N )

[0017] Among them, Atten is a learnable network layer used to learn weights for [P1, P2, ..., P N ] for weighting;

[0018] 3) Both speaker classification and style classification use relatively simple networks, using linear layers plus activation functions to predict the probability of belonging to a certain speaker or style category, and learn preliminary task-related features T speaker and T style ;

[0019] Step 4: Use mutual information to identify T speaker and T style Decoupling is performed, and mutual information reflects the correlation between two variables. Assuming there are two random variables X and Y, the mutual information MI(X; Y) between the two variables can be calculated by calculating the joint probability distribution P X,Y and the product P of two marginal probability distributions X P Y Calculated by KL divergence;

[0020] During the training of the entire downstream task, it is necessary to constrain MI(X;Y) to become smaller and smaller;

[0021] In a neural network, for two high-dimensional variables, the following steps can be used to estimate their mutual information:

[0022] 1) Initialize a random weight M;

[0023] 2) Randomly take some pairs of random variables (E i ,E′ i ), i=1,2,…,bb is the sampling size;

[0024] 3) E′ i The internal elements of the variable are randomly shuffled;

[0025] 4) Calculate the loss between the two variables as follows:

[0026]

[0027] 5) According to the calculation Update M,

[0028] 6) Repeat steps 2)-5) until M converges.

[0029] Step 5: Use the learned T speaker and T style Added to the end-to-end speech synthesis model, the end-to-end speech synthesis model adopts the encoder-attention-decoder structure. Among them, the end-to-end model is also pre-trained with large-scale data. During pre-training, T speaker and T styleThey are all vectors of all 0s. Finally, fine-tuning is performed on the multi-speaker and multi-style datasets to obtain the final model.

[0030] Beneficial effects:

[0031] The present invention proposes a self-supervised learning-based speech synthesis model, which performs unsupervised training on a large model in the speech field in a self-supervised manner, and uses two tasks of speaker recognition and style classification as downstream tasks. At the same time, mutual information technology is used for decoupling to obtain robust speaker representations and style representations, and finally they are applied to an end-to-end speech synthesis model to realize a high-quality multi-speaker and multi-style speech synthesis system.

[0032] The present invention improves the quality of multi-speaker and multi-style speech synthesis, and at the same time improves the fast adaptation ability of the synthesis model to a small amount of data. Description of the drawings

[0033] Figure 1 Upstream and downstream tasks based on self-supervised learning;

[0034] Figure 2 End-to-end multi-speaker and multi-style speech synthesis model.

[0035] Figure 3 Mel spectrogram. Detailed implementation manners

[0036] The following further illustrates the present invention in conjunction with embodiments and drawings.

[0037] The present invention proposes a speech synthesis method based on self-supervised learning and mutual information decoupling technology, which is specifically as follows:

[0038] Step 1, data preprocessing. The text needs to be preprocessed at the front end. Usually, characters are used as input, and the paired data of text and audio can be used as training data. At the same time, preprocessing also needs to extract Mel spectrogram features.

[0039] For the preprocessing of text, the natural language processing toolkit NLTK of Python is usually used to perform operations such as word segmentation and Chinese character to pinyin conversion. For the extraction of Mel spectrogram features, the common audio processing toolkit librosa of Python can be directly used. Through processes such as framing, windowing, and pre-emphasis, Mel spectrogram features are extracted from the audio.

[0040] For example, for the Chinese text "Karl Pu accompanied his grandson to play on the slide", after front-end processing, it will become "ka2er2pu3#1pei2 wai4 sun1#1wan2 hua2 ti1#3.", where #1 and #3 represent prosodic pauses. For the audio, after being processed by the librosa package, the extracted Mel spectrogram features are as Figure 3 shown.

[0041] Step 2: Use self-supervised learning to train the large model. Models such as HUBERT and wav2vec are used as pre-trained large models, and a large number of unlabeled methods are used to train them.

[0042] In step three, we design speaker classification and style classification as downstream tasks, and fix the weight parameters of the model trained in step two to obtain task-related feature representations.

[0043] Usually, when a neural network is trained, all weights in the network are updated. In this invention, when training downstream tasks, the parameters of the upstream model are frozen so that they will not change with the training of downstream tasks and are only used to extract common features, such as Figure 1 As shown in the left half of the figure, the input audio is obtained and the output of the N-layer network [P1, P2, ..., P N ].

[0044] 4) The large model in step 2 contains an N-layer neural network, and the output of the N-layer network [P1, P2, ..., P N ] will be used to learn the final task-related feature representation.

[0045] 5) Design an attention mechanism module, which is a learnable neural network layer that is used to weight the output of the N-layer neural network to obtain a temporary general feature representation P F , as shown in the following formula:

[0046] P F =Atten(P1,P2,…,P N )

[0047] Among them, Atten is a learnable network layer used to learn weights for [P1, P2, ..., P N ] for weighting.

[0048] 6) Both speaker classification and style classification use relatively simple networks, using linear layers plus activation functions to predict the probability of belonging to a certain speaker or style category, and learn preliminary task-related features T speaker and T style. When solving multi-classification problems through neural networks, the most common way is to set n output nodes in the last layer, whether in shallow neural networks or CNNs. Generally, the number of nodes in the last output layer is equal to the target number of the classification task. For this patent, the last number of nodes is the number of speakers or the number of style types. Then, for each sample, the neural network can obtain a 2-dimensional array as the output result, and each dimension in the array corresponds to a category. In the most ideal case, if a sample belongs to this speaker or style, then the output value of the output node corresponding to this category should be 1, and the outputs of other nodes are all 0, that is, [0,0,1,0,…0,0]. This array is also the label of the sample, which is the most expected output result of the neural network. The cross entropy is used to determine the degree of closeness between the actual output and the expected output.

[0049] Step 4: Use mutual information to identify T speaker and T style Decoupling is performed, and mutual information reflects the correlation between two variables. Assuming there are two random variables X and Y, the mutual information MI(X; Y) between the two variables can be calculated by calculating the joint probability distribution P X,Y and the product P of two marginal probability distributions X P Y During the training of the entire downstream task, it is necessary to constrain MI(X;Y) to become smaller and smaller.

[0050] In a neural network, for two high-dimensional variables, the following steps can be used to estimate their mutual information:

[0051] 1) Initialize a random weight M;

[0052] 2) Randomly take some pairs of random variables (E i ,E′ i ), i=1,2,…,bb is the sampling size;

[0053] 3) E′ i The internal elements of the variable are randomly shuffled;

[0054] 4) Calculate the loss between the two variables as follows:

[0055]

[0056] 5) According to the calculation Update M,

[0057] 6) Repeat steps 2)-5) until M converges.

[0058] Step 5: Use the learned Tspeaker and T style Added to the end-to-end speech synthesis model, the end-to-end speech synthesis model adopts the encoder-attention-decoder structure. The overall model is shown in the figure Figure 2 As shown in the figure. The end-to-end model is also pre-trained with large-scale data. speaker and T style are all vectors of 0, and finally fine-tune on multi-speaker and multi-style datasets to obtain the final model.

Claims

1. A speech synthesis method based on self-supervised learning and mutual information decoupling technology, characterized in that: It includes the following four steps: Step 1: Data preprocessing: Text needs to be processed on the front end, with characters as input and text and audio pairs as training data; Preprocessing also requires extraction of Mel spectrum features; Step 2: Use self-supervised learning to train the large model: Use the HUBERT and wav2vec models as pre-trained large models, and use a large number of unlabeled alignment methods for training; Step 3: Design speaker classification and style classification as downstream tasks. Fix the weight parameters of the model trained in step 2 to obtain task-related feature representations: (1) The large model in step 2 contains an N-layer neural network, and the output of the N-layer network [P1, P2, ..., P N ] are used to learn the final task-related feature representation; (2) Design an attention mechanism module, which is a learnable neural network layer that is used to weight the output of the N-layer neural network to obtain a temporary general feature representation P F , as shown in the following formula: P F =Atten(P1,P2,…,P N ) Among them, Atten is a learnable network layer used to learn weights for [P1, P2, ..., P N ] for weighting; (3) Speaker classification and style classification use linear layers plus activation functions to predict the probability of belonging to a certain speaker or style category, and learn preliminary task-related features T speaker and T style ; Step 4: Use mutual information to identify T speaker and T style Decoupling is performed, and mutual information reflects the correlation between two variables. Assuming two random variables X and Y, the mutual information MI(X; Y) between the two variables can be calculated by calculating the joint probability distribution P X,Y and the product P of two marginal probability distributions X P Y Calculated by KL divergence; In the training of the entire downstream task, MI(X;Y) needs to be constrained as an additional loss function to ensure that T speaker and T style There will be no correlation between the two; It is necessary to constrain MI(X;Y) to be smaller and smaller; Step 5: Use the learned T speaker and T style Added to the end-to-end speech synthesis model, the end-to-end speech synthesis model adopts the encoder-attention-decoder structure; Among them, the end-to-end model is first pre-trained with large-scale data. During the pre-training, T speaker and T style Finally, fine-tune the model on multi-speaker and multi-style datasets to obtain the final model.

2. The speech synthesis method based on self-supervised learning and mutual information decoupling technology according to claim 1, characterized in that: In the fourth step of the neural network, the mutual information of two high-dimensional variables is estimated by the following steps: (1) Initialize a random weight M; (2) Randomly select some pairs of random variables (E i ,E′ i ), i=1,2,…,bb is the sampling size; (3) For E′ i The internal elements of the variable are randomly shuffled; (4) Calculate the loss between two variables as follows: (5) According to the calculation Update M, Where ε is the learning rate, Δ M is a gradient change; (6) Repeat steps (2)-(5) until M converges.

Images