Adversarial and meta-learning methods based on speaker emotion speech synthesis model

Abstract

The present invention belongs to the technical field of deep learning for speech synthesis, and specifically relates to an adversarial and meta-learning method based on a speaker emotional speech synthesis model, which optimizes the emotional speech synthesis system by using adversarial training and meta-learning. It includes data preprocessing, the design of an end-to-end speech synthesis basic model, and the addition of an adversarial training module to erase the timbre information in the emotional embedding representation. The adversarial module mainly includes a gradient reversal layer and a speaker classifier and training based on meta-learning. The meta-learning method is used to improve the generalization ability of the emotional speech synthesis model, so that it can quickly adapt to a small amount of speaker speech data.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to an adversarial and meta-learning method based on a speaker's emotional speech synthesis model, which optimizes an emotional speech synthesis system by using adversarial training and meta-learning. Background Art

[0002] Speech is one of the important tools for human communication. Human speech contains not only text symbol information, but also changes in people's feelings and emotions. For example, the same sentence often has different meanings and impressions on the listener due to different emotions of the speaker. This is what is meant by "listening to the words and the sound". However, general speech synthesis systems focus more on the naturalness and accuracy of synthesized speech, ignoring the emotional factors contained in the speech signal.

[0003] In recent years, driven by deep learning and other methods, speech synthesis technology has made great progress, especially emotional speech synthesis. Because it can organically integrate spoken language analysis, emotional analysis and computer technology, it lays the foundation for realizing a people-oriented and personalized speech interaction system.

[0004] At present, most of the research at home and abroad uses an unsupervised style encoder to learn the emotional embedding of the reference audio, thereby achieving end-to-end emotional speech synthesis. However, since the emotion comes from the speech of the reference audio (source) speaker, the timbre information of the source speaker can also be transmitted to the synthesized speech, making the synthesized speech sound like the source speaker or between the source speaker and the target speaker, which will cause the so-called speaker timbre leakage problem. In addition, due to the high cost of recording emotional data, it is difficult for us to obtain a large amount of data with different emotions from any speaker, resulting in insufficient emotional expression in the synthesized speech, thereby restricting the quality of emotional speech synthesis.

[0005] In order to transfer emotions from the reference audio of the source speaker to synthesize the emotional speech of the target speaker, while maintaining the timbre of the target speaker in the synthesized speech, we proposed an emotion decoupling module based on adversarial training to erase the speaker information contained in the emotion embedding during the model training process. In addition, considering that recording a corpus with different emotion categories is very challenging, we use meta-learning to improve the generalization ability of the emotional speech synthesis model so that it can quickly adapt to a small amount of speaker speech data. Summary of the invention

[0006] In order to solve the technical problems raised in the background technology, the present invention adopts an adversarial and meta-learning method based on a speaker's emotional speech synthesis model.

[0007] The technical solution of the present invention is an adversarial and meta-learning method based on a speaker's emotional speech synthesis model, comprising the following steps:

[0008] Step 1: Data preprocessing: The text needs to be processed at the front end, and the paired data of text and audio is used as training data. Mel spectrum features also need to be extracted.

[0009] Step 2: Design of end-to-end speech synthesis basic model: Based on the end-to-end speech synthesis Tacotron2, we made the following modifications for the emotional speech synthesis task:

[0010] 1) Add speaker information. Different speakers are encoded as different speaker IDs. The speaker ID is used as input and the speaker embedding representation is obtained by looking up the LUT.

[0011] 2) Add an emotion encoder to learn emotion embedding representation. The input of the emotion encoder is the reference audio. The emotion encoder contains a 5-layer one-dimensional convolution and a bidirectional LSTM. After passing through the emotion encoder, the emotion embedding representation is obtained.

[0012] The speaker embedding representation and the emotion embedding representation are combined with the text representation output by the text encoder of the Tacotron2 model to jointly guide the generation of the final Mel-spectrogram features.

[0013] Step 3: Add an adversarial training module: Add an adversarial training module to erase the timbre information in the emotional embedding representation. The adversarial module mainly includes a gradient reversal layer and a speaker classifier.

[0014] After adding the adversarial training module, we first use data from several different speakers and emotions for pre-training to obtain a basic emotional speech synthesis model to achieve emotional speech synthesis for the speakers in the training set.

[0015] definition These are the neural network parameters at this time and serve as the initial parameters for step 4;

[0016] Step 4: Meta-learning-based training: The emotional speech synthesis model trained in step 3 is trained again using meta-learning, which includes the following steps:

[0017] 1) First, a series of meta-tasks are constructed using the multi-speaker speech synthesis database. The support set training set and query set test set of each meta-task contain K samples and Q samples of the same speaker, and are defined;

[0018] Define each sample as ,in is the text representation of the sample, is the acoustic feature of the sample, and the acoustic feature is the Mel spectrum feature;

[0019] 2) Start iterating and executing the following training process:

[0020] a. Sample any training task m, use the Support Set of task m, and optimize the emotional speech synthesis model trained in step 3 based on the learning rate of task m to update the new parameters. ; Performing an optimization means back-propagating the trained neural network weights and performing gradient descent based on the gradient to update the weights;

[0021] b. Based on an optimized , use Query Set to calculate the loss of task m = , and calculate right The gradient of

[0022] Among them, loss is the loss function of the emotional speech synthesis model trained in step 3, which refers to the acoustic features predicted by the model in this article. and the true characteristics of the sample The error between

[0023] c. Use the above gradient and multiply it by the learning rate of the meta network , update the neural network parameters to get

[0024] in, Refers to the neural network model parameters obtained after step 3, Refers to the parameters after an update;

[0025] d. Repeat the training process of above ac;

[0026] Step 5: Synthesize audio.

[0027] In the step 1, the front-end processing of the text uses the natural language processing toolkit NLTK of Python to perform word segmentation and Chinese character conversion to pinyin operations. For the preprocessing of Mel-spectrogram feature extraction, the common audio processing toolkit librosa of Python is directly used to extract Mel-spectrogram features from the audio through framing, windowing and pre-emphasis.

[0028] The step 5 is specifically as follows: after the final training of the model is completed, the following steps are sequentially performed to synthesize emotional speech;

[0029] Fine-tune the parameters of the neural network model obtained in Step 4 using a small dataset of the target speaker.

[0030] Input the text, reference audio, and the ID of the target speaker. Through model prediction, mel-spectrogram features are obtained and finally converted into audio.

[0031] Beneficial effects

[0032] 1. Build an end-to-end emotional speech synthesis system and use an unsupervised style encoder to learn the emotional embedding representation in the audio.

[0033] 2. Design an emotion decoupling module based on adversarial training to erase the speaker information contained in the emotional embedding during model training.

[0034] 3. Use meta-learning to improve the generalization ability of the emotional speech synthesis model, enabling it to quickly adapt to a small amount of speaker speech data. Description of the drawings

[0035] Figure 1 Basic model for speech synthesis.

[0036] Figure 2 Model diagram after adding the adversarial module.

[0037] Figure 3 Mel-spectrogram schematic diagram. Detailed implementation manners

[0038] The following further describes the present invention in conjunction with the drawings and embodiments.

[0039] The adversarial and meta-learning method for the speaker-based emotional speech synthesis model, the specific steps are as follows:

[0040] Step 1, preprocessing of data. The text needs to be preprocessed at the front end. Usually, characters are used as input, and the paired data of text and audio can be used as training data. At the same time, the extraction of mel-spectrogram features also needs to be carried out during preprocessing.

[0041] For the preprocessing of text, the natural language processing toolkit NLTK in Python is usually used to perform operations such as word segmentation and Chinese character to pinyin conversion. For the extraction of mel-spectrogram features, the common audio processing toolkit librosa in Python can be directly used. Through processes such as framing, windowing, and pre-emphasis, mel-spectrogram features are extracted from the audio.

[0042] For example, for the Chinese text "Karl Pu accompanied his grandson to play on the slide", after preprocessing, it becomes "ka3 er3 pu3 #1pei1 wai4 sun1 #1 wan2 hua1 ti1 #3", where #1 and #3 represent different lengths of pause times.

[0043] For the audio corresponding to the text, the Mel spectrum obtained after preprocessing is as follows Figure 3 shown.

[0044] Step 2: Design of end-to-end speech synthesis basic model. The present invention is designed based on the end-to-end speech synthesis Tacotron2. The function of Tacotron2 is to input text and predict the Mel spectrum features. For the emotional speech synthesis task, we made the following modifications:

[0045] 1) Add speaker information. Different speakers are encoded as different speaker IDs. The speaker ID is used as input and the speaker embedding representation is obtained by looking up the LUT.

[0046] For example, taking three speakers as an example, the three speakers can be input in the following form:

[0047] Speaker_1 = [1,0,0]

[0048] Speaker_2 = [0,1,0]

[0049] Speaker_3 = [0,0,1]

[0050] 2) Add an emotion encoder to learn emotion embedding representation. The input of the emotion encoder is the reference audio. The emotion encoder contains a 5-layer one-dimensional convolution and a bidirectional LSTM. After passing through the emotion encoder, the emotion embedding representation will be obtained.

[0051] The speaker embedding representation and the emotion embedding representation are combined with the text representation output by the text encoder of the Tacotron2 model to jointly guide the generation of the final Mel-spectrogram features, such as Figure 1 shown.

[0052] Step 3: Add an adversarial training module. Considering that the emotional embedding representation may contain some speaker information, which will interfere with the timbre of the final synthesized speech, an adversarial training module is added to erase the timbre information in the emotional embedding representation. The adversarial module mainly includes a gradient reversal layer and a speaker classifier, such as Figure 2 As shown in the dashed box.

[0053] After adding the adversarial training module, you can first use data from several different speakers and emotions for pre-training to obtain a basic emotional speech synthesis model, which can realize emotional speech synthesis for the speakers in the training set.

[0054] definition These are the neural network parameters at this time and serve as the initial parameters for step 4.

[0055] Step 4: Meta-learning-based training. In order to improve the generalization ability of the model and reduce the model's dependence on data, the model trained in step 3 is trained again using meta-learning. Specifically, the following steps are included:

[0056] First, a series of meta-tasks are constructed using the multi-speaker speech synthesis database. The support set (training set) and query set (test set) of each meta-task contain K samples and Q samples of the same speaker, and are defined;

[0057] Define each sample as ,in is the text representation of the sample, is the acoustic feature of the sample, and the acoustic feature is the Mel spectrum feature;

[0058] Next, start iterating through the following training process:

[0059] a. Sample any training task m, use the Support Set of task m, optimize the network based on the learning rate of task m, and update the new parameters . Performing an optimization means backpropagating the trained neural network weights once and performing gradient descent based on the gradient to update the weights.

[0060] b. Based on an optimized , use Query Set to calculate the loss of task m = , and calculate right The gradient of . Among them, loss is the loss function of the emotional speech synthesis model trained in step 3, which refers to the acoustic features predicted by the model in this article. and the true characteristics of the sample The error between .

[0061] c. Use the above gradient and multiply it by the learning rate of the meta network , update the parameters of the original network to get ,in, ϕ Refers to the neural network model parameters obtained after step 3, Refers to the parameters after an update.

[0062] d. Repeat the training process of ac above. (Depending on the size of the training data set, for example, if the training data set contains only 100 data, iterate all meta-task sets about 20 times and then stop)

[0063] Step 5: Synthesize audio. After the final training of the model is completed, the following steps can be performed in sequence to synthesize emotional speech.

[0064] 1. Get the network parameters through step 4, and use a small amount of target speaker data set to perform final fine-tuning on the parameters.

[0065] When the timbre of a specified speaker needs to be synthesized, the audio resources of the specified speaker may only be 3-5 minutes. Therefore, this small amount of data needs to be used to perform final fine-tuning on the above network.

[0066] 2. Input text, reference audio (consistent with the target emotion), and the target speaker ID. The model predicts the mel-spectrogram features and converts them into audio.

Claims

1. The adversarial and meta-learning method based on the speaker emotion speech synthesis model is characterized by: The steps include: Step 1: Data preprocessing: The text needs to be processed at the front end, and the paired data of text and audio is used as training data. Mel spectrum features also need to be extracted. Step 2: Design of end-to-end speech synthesis basic model: Design based on end-to-end speech synthesis Tacotron2, and modify the emotional speech synthesis task as follows: 1) Add speaker information. Different speakers are encoded as different speaker IDs. The speaker ID is used as input and the speaker embedding representation is obtained by looking up the LUT. 2) Add an emotion encoder to learn emotion embedding representation. The input of the emotion encoder is the reference audio. The emotion encoder contains a 5-layer one-dimensional convolution and a bidirectional LSTM. After passing through the emotion encoder, the emotion embedding representation is obtained. The speaker embedding representation and the emotion embedding representation are combined with the text representation output by the text encoder of the Tacotron2 model to jointly guide the generation of the final Mel-spectrogram features. Step 3: Add an adversarial training module: Add an adversarial training module to erase the timbre information in the emotional embedding representation. The adversarial module contains a gradient reversal layer and a speaker classifier. After adding the adversarial training module, we first use data from several different speakers and emotions for pre-training to obtain a basic emotional speech synthesis model to achieve emotional speech synthesis for the speakers in the training set. definition These are the neural network parameters at this time and serve as the initial parameters for step 4; Step 4: Meta-learning-based training: The emotional speech synthesis model trained in step 3 is trained again using meta-learning, which includes the following steps: 1) First, a series of meta-tasks are constructed using the multi-speaker speech synthesis database. The support set training set of each meta-task contains K samples and Q samples of the same speaker, and the query set test set contains K samples and Q samples of the same speaker, and is defined; Define each sample as ,in is the text representation of the sample, is the acoustic feature of the sample, and the acoustic feature is the Mel spectrum feature; 2) Start iterating and executing the following training process: a samples any training task m, uses the Support Set of task m, and based on the learning rate of task m, Perform an optimization and update the new parameters ; Performing an optimization means back-propagating the trained neural network weights and performing gradient descent based on the gradient to update the weights; b is based on an optimized , use Query Set to calculate the loss of task m = , and calculate right The gradient of Among them, loss is the loss function of the emotional speech synthesis model trained in step 3, specifically referring to the acoustic features predicted by the model and the true characteristics of the sample The error between c uses the above gradient and multiplies it by the learning rate of the meta network , update the neural network parameters to get ; in, Refers to the neural network model parameters obtained after step 3, Refers to the parameters after an update; d. Repeat the training process from a to c above; Step 5: Synthesize audio.

2. The adversarial and meta-learning method based on the speaker emotion speech synthesis model according to claim 1, characterized in that: In the step 1, the front-end processing of the text uses the natural language processing toolkit NLTK of Python to perform word segmentation and Chinese character conversion to pinyin operations. For the extraction of audio features, the commonly used audio processing toolkit librosa of Python is directly used to extract Mel spectrum features from the audio through framing, windowing and pre-emphasis.

3. The adversarial and meta-learning method based on the speaker emotion speech synthesis model according to claim 1, characterized in that: The step 5 specifically includes: after completing the final training of the model, sequentially executing the following steps to synthesize emotional speech; Use a small amount of target speaker data set to perform final fine-tuning on the parameters of the neural network model obtained in step 4; Input text, reference audio, and the target speaker's ID, obtain the mel-spectrogram features through model prediction, and finally convert them into audio.

Images