A method for generating a mongolian emotional speech synthesis based on a conditional generative adversarial network

Abstract

A Mongolian emotional speech synthesis method based on a conditional generative adversarial network, converts Mongolian text from letters to phonemes to obtain phoneme sequence data; extract the mel spectrum I from the Mongolian emotional audio; input the phoneme sequence data and the mel spectrum I into the speech text joint pre-training model to train the alignment information of the speech and the text; add a sentiment module to the FastSpeech2 acoustic model to obtain an improved model, use the improved model as a generator, introduce a discriminator and a sentiment label to construct a conditional generative adversarial network; the generator takes the output of the speech text joint pre-training model as input and outputs the mel spectrum II; the discriminator judges the mel spectrum II and the mel spectrum I until the error is less than the preset; input the mel spectrum II meeting the error requirement into the vocoder to convert the acoustic features into speech waveforms, and complete the Mongolian emotional speech synthesis. The present application can directly synthesize emotional speech from characters, greatly reducing the requirement for linguistic knowledge.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and relates to speech synthesis, in particular to a Mongolian emotional speech synthesis method based on a conditional generative adversarial network. BACKGROUND

[0002] Speech synthesis technology refers to a technology of converting any text into fluent speech by computer analysis and processing. Speech synthesis researches at home and abroad have experienced the development process of source-filter-based speech synthesis, waveform splicing-based speech synthesis, statistical parameter-based speech synthesis, and deep neural network-based speech synthesis methods.

[0003] Mongolian speech with high naturalness and intelligibility can be synthesized, but Mongolian emotional speech synthesis is still in its infancy due to the problems of lack of emotional corpus and less research investment, so there is a huge space for development in the synthesis of Mongolian emotional speech based on deep neural networks. SUMMARY

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a Mongolian emotional speech synthesis method based on a conditional generative adversarial network, which has an end-to-end generative text-to-emotional speech model and can directly synthesize emotional speech from characters, greatly reducing the requirement for linguistic knowledge.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0006] A Mongolian emotional speech synthesis method based on a conditional generative adversarial network, comprising:

[0007] Step 1, obtaining a Mongolian text to be synthesized and a Mongolian emotional audio;

[0008] Step 2, converting the Mongolian text into phoneme sequence data by letter-to-phoneme conversion; and extracting a mel spectrum 1 from the Mongolian emotional audio;

[0009] Step 3, inputting the phoneme sequence data and the mel spectrum 1 into a speech-text joint pre-training model to train the alignment information of speech and text;

[0010] Step 4, adding an emotional module to a FastSpeech2 acoustic model to obtain an improved model, taking the improved model as a generator, introducing a discriminator and an emotional label to construct a conditional generative adversarial network; the generator takes the output of the speech-text joint pre-training model as input and outputs a mel spectrum 2, and the discriminator judges the mel spectrum 2 and the mel spectrum 1 until the error is less than a preset value;

[0011] Step 5, the mel-spectrogram that meets the error requirement is input to the vocoder, the acoustic feature is converted into a speech waveform, and the Mongolian emotional speech synthesis is completed.

[0012] Compared with the prior art, the innovation of the present application lies in that:

[0013] The Mongolian text is converted into phonemes using an Encoder-Decoder model, which can be seamlessly integrated into an end-to-end TTS system, and a conditional generative adversarial network is used to increase the emotion encoder module FastSpeech2 model as an acoustic model, and the implicit emotion style features are learned in combination with the emotion label condition, and the phonemes are directly generated into mel-spectrogram, and the HiFiGAN vocoder directly converts the mel-spectrogram into a speech waveform. The present application takes the conditional generative adversarial network as the core structure, combines the FastSpeech2 acoustic model and the HiFiGAN vocoder, realizes the Mongolian emotional speech synthesis, and this synthesis method has not been applied in the Mongolian emotional speech synthesis. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is the flowchart of the present application.

[0015] Figure 2 is the schematic diagram of the generative adversarial network.

[0016] Figure 3 is the schematic diagram of the FastSpeech model.

[0017] Figure 4 is the schematic diagram of the FastSpeech2 model.

[0018] Figure 5 is the schematic diagram of the FastSpeech2 model with emotion encoder.

[0019] Figure 6 is the schematic diagram of the emotion encoder module structure.

[0020] Figure 7 is the schematic diagram of the SpeechLM speech text joint pre-training model. DETAILED DESCRIPTION

[0021] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and examples.

[0022] As Figure 1 shown, the present application is a Mongolian emotional speech synthesis method based on a conditional generative adversarial network, comprising the following steps:

[0023] Step 1, obtaining the Mongolian text to be synthesized and the Mongolian emotional audio.

[0024] In the present application, a parallel corpus composed of Mongolian text and Mongolian emotional audio is used, and in the parallel corpus, each Mongolian text corresponds to a Mongolian audio with an emotional label.

[0025] Step 2: Perform letter-to-sound conversion on the Mongolian text to obtain phoneme sequence data. Extract the Mel-spectrogram from the Mongolian emotional audio.

[0026] This step includes the acquisition of phoneme sequence data and the extraction of Mel-spectrogram. For phoneme sequence data, the present application uses a Mongolian G2P (Grapheme-to-Phoneme) conversion model based on Encoder-Decoder for letter-to-sound conversion. The Encoder-Decoder structure has shown the latest results in different NLP tasks. The main idea of these methods has two steps: the first step is to map the input sequence to a vector; the second step is to generate the output sequence representation based on the learning vector. The Encoder-Decoder model generates an output after the encoder processes the complete input sequence, which allows the decoder to learn from any part of the input without being limited by a fixed context window. For example, first convert the Mongolian word: (Latin transcription: "cqhihv", the encoder input is the "cqhihv" grapheme sequence, and the decoder produces "q oe h ws h" as the phoneme sequence. The model stops predicting after generating the phoneme end marker.

[0027] The present application uses an Encoder-Decoder architecture, the encoder uses a convolutional neural network structure with residual connection and batch normalization. The decoder is a bidirectional LSTM that maps the input Mongolian text sequence to a vector and generates the output sequence representation based on the learning vector, i.e. phoneme sequence data. Since the letter-to-sound task is actually a simplified machine translation task, the sequence-to-sequence model is fully suitable for handling letter-to-sound problems. The neural network-based G2P conversion has strong robustness to spelling errors and out-of-vocabulary words; it has good generalization ability. In addition, it can also be seamlessly integrated into an end-to-end TTS system.

[0028] Further, the input Mongolian text may contain special characters such as dates, numbers, English letters, and other non-Mongolian characters. Before performing Mongolian-to-Latin conversion, the special characters need to be transcribed into the corresponding Mongolian expression and corrected, otherwise it will cause word omission, Mongolian-to-Latin conversion errors, etc. When performing Latin conversion, the Mongolian text is uniformly transcribed into Latin representation sequence according to the corresponding relationship between Mongolian and Latin.

[0029] For the mel-spectrogram one, the invention uses librosa to process the Mongolian emotional audio signal, first, the audio signal is pre-emphasized, framed and windowed. Second, the short-time Fourier transform STFT is performed on each frame of signal to obtain the short-time amplitude spectrum. Finally, the short-time amplitude spectrum is obtained by the mel filter bank to obtain the mel-spectrogram one.

[0030] Step 3, input the phoneme sequence data and the mel-spectrogram one into the speech-text joint pre-training model to train the alignment information of speech and text.

[0031] In this step, the speech-text joint pre-training model based on SpeechLM is used, after training, the speech and text can be better aligned, which helps to improve the performance of the downstream speech synthesis task, and the model structure is as shown in Figure 7 .

[0032] Step 4, add an emotion module to the FastSpeech2 acoustic model to obtain an improved model, use the improved model as a generator, introduce a discriminator and an emotion label to construct a conditional generative adversarial network. The generator takes the output of the speech-text joint pre-training model (phoneme sequence and alignment information of speech and text) as input and outputs the mel-spectrogram two, the discriminator judges the mel-spectrogram two and the mel-spectrogram one until the error is less than the preset.

[0033] The structure of the generative adversial network (GAN) is as shown in Figure 2 , which is actually a combination of two networks: the generator (Generator) is responsible for generating simulated data; the discriminator (Discriminator) is responsible for judging whether the input data is real or generated. The generator continuously optimizes the generated data to make the discriminator judge it, and the discriminator also optimizes itself to make itself more accurate, and the relationship between the two forms a confrontation.

[0034] The following is the objective function of the traditional GAN, in the training process of the generator and the discriminator, the objective function of the model is a two-player minimax game:

[0035]

[0036] The conditional generative adversarial network (CGAN) is a deformation of the original GAN, the generator and the discriminator both increase the additional information C as a condition, the condition can be category information or other modal data, in the invention, the emotion label is used as the emotion condition. By feeding the additional information C to the discriminator model and the generator model as part of the input layer, the CGAN is realized.

[0037] In the generative model, the prior input noise p(z) and the conditional information y are jointly generated, and the objective function of the CGAN is a binary minimax game with conditional probability:

[0038]

[0039] In the present application, based on the conditional generative adversarial network, the improved FastSpeech2 is used as the generator to realize Mongolian emotional speech synthesis.

[0040] Before the FastSpeech series acoustic model, the acoustic model has the following problems:

[0041] 1) Slow inference speed, previously, whether the acoustic model based on RNN, CNN or Transformer system, when outputting mel features, it will go through an autoregressive production process, so it will be slow;

[0042] 2) The generated speech is not robust, and the error will be accumulated in the autoregressive process, so the generated mel spectrum will have the phenomenon of repetition or omission of words;

[0043] 3) The generated speech is uncontrollable.

[0044] Considering the monotonic alignment between text and speech, in order to speed up the generation of mel spectrum, Microsoft proposed the FastSpeech model, as shown in the structure Figure 3 , based on the self-attention feedforward network in Transformer and one-dimensional convolution, taking the text (phoneme) sequence as input, non-autoregressively generating mel spectrum.

[0045] Since one phoneme can correspond to multiple mel spectrum sequences, the mel sequence generated by the general model is much longer than the input phoneme sequence, which causes the length of the input side and the output side to be extremely mismatched. Therefore, a duration model prediction module is introduced in the model to predict the duration of a phoneme, thereby making up for the difference between the two.

[0046] Therefore, the FastSpeech series has the following advantages:

[0047] 1) Non-autoregressive decoding, parallel generation of mel features, compared with autoregressive models, the calculation speed is increased;

[0048] 2) The duration model ensures one-to-one correspondence between phonemes and mel features, greatly reducing the previous model's reading omission phenomenon;

[0049] 3) The introduction of the duration model makes the generated audio controllable.

[0050] Although FastSpeech has achieved faster generation speed and similar speech quality than autoregressive models such as Tacotron as a non-autoregressive TTS model, FastSpeech still has some shortcomings, such as:

[0051] 1) It is very time-consuming to use an autoregressive TTS model as a teacher training model;

[0052] 2) Using the knowledge distillation method to train the model will cause information loss, affecting the sound quality of the synthesized speech.

[0053] FastSpeech2 improves these problems, first of all, it abandons the teacher-student training of knowledge distillation, and adopts the method of training directly on the ground-truth. Secondly, more inputs that can control the speech are introduced into the model, which includes not only the phoneme duration mentioned in FastSpeech, but also new quantities such as energy and pitch. The FastSpeech2 model structure is as shown in Figure 4 .

[0054] In the present application, an emotion encoder module is added to the FastSpeech2 model encoder, and the emotion encoder is used to realize implicit emotion style feature learning in Mongolian emotion speech. The overall structure of the modified FastSpeech2 model is as shown in Figure 5 .

[0055] Specifically, the emotion module of the present application includes a multi-layer convolutional network and an emotion classifier, the multi-layer convolutional network is used to extract acoustic features of audio, and the emotion classifier is used to predict emotion classification according to the acoustic features.

[0056] The emotion module is added to the encoder of the FastSpeech2 acoustic model as a separate subnetwork, which is used to obtain the latent feature information of different categories of emotions, and the feature information is spliced with the output of the phoneme encoder in the FastSpeech2 acoustic model, and is input into the variance adapter of the FastSpeech2 acoustic model. The input of the emotion module is the mel spectrum of the Mongolian emotion speech and the emotion label, and the output is the latent style feature vector of different categories of emotions.

[0057] Specifically, in an embodiment of the present application, the structure of the emotion module is similar to the encoder, and the multi-layer convolutional network is six convolutional layers. The emotion classifier includes a GRU layer (using the final GRU state) and two fully connected layers, and the GRU layer and the two fully connected layers are used as the emotion classifier, that is, first six convolutional layers, then a GRU layer, and finally two fully connected layers, and the overall structure is as shown in Figure 6The recognition accuracy of the predicted emotion classification and the actual emotion classification can be calculated in the model training stage, the training parameters can be reasonably adjusted, and the model performance can be improved. Deep neural networks can effectively extract more rich and complex features, and can also improve the accuracy of the model.

[0058] The emotion labels of the application include joy, anger, sadness, surprise, fear, disgust, and neutral, as additional information conditions. Thus, Mongolian speech with joy, anger, sadness, surprise, fear, disgust, and neutral can be synthesized accordingly, and the expressiveness of Mongolian speech can be enhanced.

[0059] The current end-to-end speech synthesis first generates acoustic features such as mel spectrum, linear spectrum, etc. by an acoustic model, and the application will use an acoustic model based on FastSpeech2 to predict the mel spectrum. By adding an emotion module to the FastSpeech2 acoustic model, implicit emotional style features in Mongolian emotional speech are learned. Based on the conditional generative adversarial network, the improved FastSpeech2 is used as a generator to realize Mongolian emotional speech synthesis. On the basis of introducing emotional conditions, the emotional speech data is trained, the generator of GAN generates mel spectrum, and the discriminator needs to constantly improve the discrimination ability to distinguish the authenticity of the generated results. In the process of confrontation, the generation effect of the model is improved to learn the best Mongolian emotional style features. The model is trained on pre-normalized phoneme sequences, which can greatly reduce the problem of incorrect pronunciation, and can be trained in parallel. There is no recursive connection between the frameworks, which can improve the performance of Mongolian speech synthesis, speed up the training speed, and make the audio process more smooth.

[0060] Step 5: input the mel spectrum that meets the error requirement into the vocoder to convert the acoustic features into speech waveform, and complete the Mongolian emotional speech synthesis.

[0061] The vocoder is a model that converts acoustic features into speech waveform. The current end-to-end speech synthesis first generates acoustic features such as mel spectrum, linear spectrum, etc. by an acoustic model, and then converts them into the final speech by the vocoder. The prosody and expressiveness in the generated speech are determined by the acoustic model, while the intelligibility is determined by the vocoder. The vocoder limits the sound quality of the final synthesized speech, and is also the computational bottleneck of the entire speech synthesis model. In order to increase the receptive field, convolutional sequence models generally need to stack very deep network layers. Sequence models have good results in the tasks of text and speech fields, but how to sample more quickly is still a relatively difficult problem.

[0062] Compared with WaveNet, which increases the receptive field by stacking dilated convolution, generating sample points one by one, the sound quality is indeed very good, but it also makes the model larger and the inference speed slower. HiFiGAN is a vocoder commonly used in academia and industry in recent years, which can convert the spectrum generated by the acoustic model into high-quality audio. This vocoder uses a generative adversarial network (GAN) as the basic generation model.

[0063] The generator of HiFiGAN mainly has two parts, one is the upsampling structure, which is composed of one-dimensional transpose convolution; the other is the multi-receptive field fusion (MRF) module, which is mainly responsible for optimizing the sampling points obtained by upsampling, which is composed of a residual network, and alternately uses dilated convolution and ordinary convolution to increase the receptive field, ensuring the quality of synthesized sound while improving the inference speed.

[0064] The discriminator of HiFiGAN has two, multi-scale (MSD) and multi-period discriminator (MPD), which respectively identify the speech from two different angles. The multi-scale discriminator is derived from the method of MelGAN vocoder, which continuously averages the speech sequence, and gradually halves the length of the speech sequence, then applies several layers of convolution on different scales of the speech, and finally flattens it as the output of the multi-scale discriminator; the multi-period discriminator is to fold the one-dimensional audio sequence into a two-dimensional plane with different sequence lengths, and apply two-dimensional convolution on the two-dimensional plane. The ultimate goal is to enhance the ability of the GAN discriminator to distinguish between synthesized and real audio as much as possible.

[0065] In this step, HiFiGAN is selected as the vocoder. The prosody, expressiveness, etc. in the generated speech are determined by the acoustic model, while the intelligibility is determined by the vocoder, which limits the sound quality of the final synthesized speech, and is also the computational bottleneck of the entire speech synthesis model. HiFiGAN can effectively improve the inference speed while ensuring the quality of the synthesized sound. The HiFiGAN model can be used as a vocoder for FastSpeech2 to synthesize audio and optimize the speed of speech synthesis.

[0066] The following provides a specific case of synthesizing Mongolian emotional speech using the method of the application, which is mainly divided into two stages. In the training stage, a laboratory-recorded Mongolian emotional speech dataset with seven types of emotions (the dataset includes Mongolian text, Mongolian audio and emotion labels) is used for multiple training, and the game confrontation training based on the conditional generative adversarial network enables the generator to learn the acoustic features output in the acoustic model well and continuously improve the ability to generate mel spectrum; in the inference stage, to synthesize speech with a certain emotion, only the Mongolian text and emotion label need to be input into the generator, and after obtaining the mel spectrum, the HiFiGAN vocoder is used to synthesize Mongolian speech with the corresponding emotion.

[0067] In summary, the main flow of the application includes three parts: front-end processing, predicted mel spectrum module and mel spectrum conversion speech waveform module. The front-end processing part converts special characters, corrects Latin, and first converts the input Mongolian text into special character transcription, then corrects the text after special character conversion, and finally converts Latin to reduce the difficulty of the traditional Mongolian letter-to-phoneme method. After obtaining the Latin phoneme sequence, it is input into the speech text joint pre-training model to obtain the alignment information of speech and text; the predicted mel spectrum model part is to input the Latin phoneme sequence and the alignment information of speech and text into the FastSpeech2 acoustic model with emotion encoder to obtain the corresponding mel spectrum. Compared with the traditional TTS, the prosody and expressiveness of the generated speech are better, and the emotional features are more abundant. Finally, the mel spectrum is synthesized into Mongolian emotional speech by the HiFiGAN vocoder, which guarantees the intelligibility of the generated speech while optimizing the speed of speech synthesis.

Claims

1. A method for synthesizing Mongolian emotional speech based on conditional generative adversarial networks, characterized in that, include: Step 1: Obtain the Mongolian text and Mongolian emotional audio to be synthesized; Step 2: Convert the Mongolian text from letters to phonemes to obtain phoneme sequence data; extract Mel spectrum from the Mongolian emotional audio. Step 3: Train the alignment information of speech and text by combining the phoneme sequence data with the Mel spectrum input speech-text joint pre-training model; Step 4: Add an emotion module to the FastSpeech2 acoustic model to obtain an improved model. Use the improved model as a generator, and introduce a discriminator and emotion labels to construct a conditional generative adversarial network. The generator takes the output of the speech-text joint pre-trained model as input and outputs Mel spectrum II. The discriminator judges the Mel spectrum II and Mel spectrum I until the error is less than a preset value. The emotion module is added as a separate sub-network to the encoder of the FastSpeech2 acoustic model to obtain the potential feature information of different emotion categories. The feature information is concatenated with the output of the phoneme encoder in the FastSpeech2 acoustic model and input together into the variance adapter of the FastSpeech2 acoustic model. Step 5: Input the Mel-spectrum 2 that meets the error requirements into the vocoder to convert the acoustic features into speech waveforms and complete the Mongolian emotional speech synthesis.

2. The Mongolian emotional speech synthesis method based on conditional generative adversarial networks according to claim 1, characterized in that, In step 1, the Mongolian text and the Mongolian emotional audio are parallel corpora, with each Mongolian text corresponding to a Mongolian audio with an emotional tag.

3. The Mongolian emotional speech synthesis method based on conditional generative adversarial networks according to claim 1, characterized in that, In step 2, a Mongolian G2P conversion model based on Encoder-Decoder is used to convert letters to phonemes. The encoder uses a convolutional neural network structure with residual connections and batch normalization, and the decoder is a dual LSTM that maps the input Mongolian text sequence to a vector. Based on the learned vector, the output sequence representation, i.e., the phoneme sequence data, is generated.

4. The Mongolian emotional speech synthesis method based on conditional generative adversarial networks according to claim 1, characterized in that, In step 2, the Mongolian emotional audio is processed using librosa. First, the audio signal is pre-emphasized, framed, and windowed. Second, a short-time Fourier transform (STFT) is performed on each frame to obtain the short-time amplitude spectrum. Finally, the short-time amplitude spectrum is processed through a Mel filter bank to obtain the Mel spectrum.

5. The Mongolian emotional speech synthesis method based on conditional generative adversarial networks according to claim 1, characterized in that, Step 3 involves using a joint pre-trained speech-text model based on SpeechLM.

6. The Mongolian emotional speech synthesis method based on conditional generative adversarial networks according to claim 1, characterized in that, In step 4, the emotion module includes a multi-layer convolutional network and an emotion classifier; the multi-layer convolutional network extracts acoustic features of the audio, and the emotion classifier predicts the emotion classification based on the acoustic features.

7. The Mongolian emotional speech synthesis method based on conditional generative adversarial networks according to claim 6, characterized in that, The multi-layer convolutional network consists of six convolutional layers; the sentiment classifier includes one GRU layer and two fully connected layers.

Images