A Mongolian speech synthesis method based on Conformer and MelGAN

Abstract

A Mongolian speech synthesis method based on Conformer and MelGAN, which obtains Mongolian text data, extracts text features, converts the text feature encoding into a phoneme sequence, and inputs the phoneme sequence into a BERT pre-training model; uses the BERT pre-training model to train Mongolian phoneme information, learns the rhythmic features in pronunciation, and aligns the phoneme information; uses a FastSpeech2 acoustic model based on a lightweight Conformer module to generate acoustic features, i.e., a Mel spectrogram, in parallel through phoneme information; uses a GAN-based lightweight model MelGAN as a vocoder, adds Mongolian speech features therein, and converts the Mel spectrogram into a speech waveform by inference, thereby realizing Mongolian speech synthesis. The present invention can improve the speed of Mongolian speech synthesis and the naturalness of synthesized speech.

Description

Technical Field

[0001] The invention belongs to the technical field of artificial intelligence, relates to a speech synthesis technology, and particularly relates to a Mongolian speech synthesis method based on Conformer and MelGAN. Background Art

[0002] Speech Synthesis is also known as Text to Speech (TTS). It is a technology that uses computers to convert any input text into realistic speech through computational reasoning. Speech synthesis research at home and abroad has undergone a variety of technological evolutions. Due to the rapid development of artificial intelligence technology, speech synthesis methods based on deep learning have made new breakthroughs in the field of speech synthesis.

[0003] With the development of science and technology, the intelligibility and naturalness of synthesized speech have been greatly improved. At present, the speech synthesis technology of mainstream languages ​​(such as Chinese and English) is quite mature, and the synthesized speech is close to the real speech. However, there is still a lot of room for exploration in the speech synthesis technology of Mongolian and other minority languages. Due to the scarcity of corpus and other problems, the speech quality and synthesis speed of minority languages ​​such as Mongolian are obviously insufficient, and the relevant speech synthesis research is still in its early stages. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art mentioned above, the object of the present invention is to provide a Mongolian speech synthesis method based on Conformer and MelGAN to improve the speed of Mongolian speech synthesis and the naturalness of synthesized speech.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A Mongolian speech synthesis method based on Conformer and MelGAN comprises the following steps:

[0007] Step 1, obtaining Mongolian text data, extracting text features, converting the text feature encoding into a phoneme sequence, and inputting the phoneme sequence into a BERT pre-training model;

[0008] Step 2: Use the BERT pre-training model to train Mongolian phoneme information, learn the rhythmic features in pronunciation, and align the phoneme information;

[0009] Step 3: Using the FastSpeech2 acoustic model based on the lightweight Conformer module, acoustic features, namely the Mel spectrogram, are generated in parallel through phoneme information;

[0010] Step 4: Use the GAN-based lightweight model MelGAN as a vocoder, add Mongolian speech features to it, and convert the Mel spectrogram into a speech waveform by reasoning, thereby realizing Mongolian speech synthesis.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] The present invention firstly performs feature pre-training on Mongolian text through the Bert model to enhance the correlation between text and phonemes. Then, the phonemes are directly generated into Mel spectra using a Conformer-based acoustic model, which simplifies the model complexity and improves the naturalness of speech. Finally, the Mel spectra are directly converted into speech waveforms through the MelGAN vocoder, which greatly improves the speed of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the process of the present invention.

[0014] Figure 2 This is a schematic diagram of the BERT pre-training model of the present invention

[0015] Figure 3 It is a schematic diagram of the lightweight Conformer module of the present invention.

[0016] Figure 4 It is a schematic diagram of the feature prediction model of the present invention (i.e., a schematic diagram of each prediction module in the variable adapter).

[0017] Figure 5 It is an overall schematic diagram of the acoustic characteristic model of the present invention.

[0018] Figure 6 Schematic diagram of the lightweight model MelGAN of the present invention. DETAILED DESCRIPTION

[0019] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0020] The present invention is a Mongolian speech synthesis method based on Conformer and MelGAN, which mainly includes four links: front-end processing, phoneme pre-training, Mel spectrum prediction and Mel spectrum conversion speech. The present invention uses a Seq2Seq model to convert text into a phoneme feature sequence, performs feature pre-training based on a BERT pre-training model, enhances the correlation between text and phonemes, learns Mongolian rhythm and phoneme alignment information, and completes phoneme pre-training; and based on the acoustic model of Conformer, extracts acoustic features according to the phoneme sequence to generate a feature spectrum, and completes Mel spectrum prediction; finally, the model based on MelGAN is used as a vocoder to convert the feature spectrum to a speech waveform, and the Mel spectrum is converted into a speech waveform to realize speech synthesis. The present invention can be seamlessly integrated into various text-to-speech systems, reduce the number of parameters, reduce the demand for hardware configuration, improve the speed of speech synthesis, and can synthesize natural and fluent speech, which is suitable for speech synthesis of small languages.

[0021] The following is a specific process of an embodiment of the present invention, and the steps are described as follows:

[0022] Step 1: obtain Mongolian text data, extract text features, convert the text feature encoding into a phoneme sequence, and input the phoneme sequence into the BERT pre-training model.

[0023] For example, in the present invention, due to the characteristics of Mongolian text forms, Mongolian text is first converted into corresponding character codes, then uniformly converted into Latin characters after processing, and finally the phoneme sequence of Mongolian text is obtained through the grapheme to phoneme model (G2P). The phoneme sequence consists of multiple phonemes. For example, the Mongolian " ” (Chinese meaning: hit) extracts text features and is expressed as: “cqhihv”. It is converted into phonemes through encoding and is expressed as: “q oe h ws h”.

[0024] Step 2: Use the BERT pre-training model to train Mongolian phoneme information, learn the prosodic features in pronunciation, and align the phoneme information. The mechanism of the BERT pre-training model can provide phoneme alignment information for the subsequent steps of the present invention using a non-autoregressive method, supplement the prosodic features missing in the phoneme information, and finally realize parallel output of acoustic features.

[0025] In an embodiment of the present invention, the BERT pre-training model is an encoder composed of a stack of Conformer modules in series. The Conformer module is a single Conformer structure, which can calculate the dependency of related phonemes through a self-attention mechanism, so that the relevance of phoneme features is closer. In order to optimize the phoneme sequence in the non-autoregressive Mongolian language synthesis model and supplement the prosodic features in the phoneme sequence, the present invention adds additional Mongolian feature information to the BERT pre-training model.

[0026] Specifically, the BERT pre-training model of the present invention is as follows: Figure 2 As shown, its input consists of two segments, which are two different representations of the same content. The first segment is a phoneme sequence, which contains a single International Standard (IPA) phoneme as the first marker. The second segment is a character sequence, which contains a single English character as the second marker. There are two special markers in the input sequence, namely CLS and SEP. Among them, CLS is attached before the first segment, and it can be used for additional classification targets besides the MLM (masked language model) target. Another special marker SEP is attached in two places after the first segment and the second segment respectively to indicate the segment boundary.

[0027] In the present invention, the syllable information and morpheme information of Mongolian are taken as input together, and position encoding is performed through a sine position function. In spoken Mongolian, syllables and phonemes are the basic units of speech, and morphemes and characters are the basic units of text. The association information of phonemes, syllables, characters, and morphemes is constructed through phoneme sequences and character sequences. The association information of the four ensures the alignment information between the phoneme level and the word level. Syllable and morpheme features provide additional rhythm and enrich the naturalness of speech. Figure 2 In the text, P refers to a single phoneme and C refers to a single character. Both P and C are different representations of the same text. Each syllable is composed of one or more phonemes, while morphemes are composed of one or more characters.

[0028] Step 3: Use the FastSpeech2 acoustic model based on the lightweight Conformer module to generate acoustic features, namely the Mel spectrogram, in parallel through phoneme information. Compared with traditional TTS, it has fewer parameters and generates better rhythm and naturalness of speech.

[0029] The FastSpeech2 acoustic model includes a stacked convolutional feedforward network and a variable adapter. Although the FastSpeech2 model generates Mel spectrograms in a non-autoregressive form, the model has a large number of parameters and has high requirements for hardware equipment.

[0030] The present invention uses a lightweight Conformer module to replace the convolutional feedforward network in the encoder and decoder of the existing FastSpeech2 model. That is, the FastSpeech2 acoustic model adopted by the present invention includes an embedding layer, an encoder, a variable adapter, a decoder and a post-processing network, and the encoder and the decoder are both composed of a stack of lightweight Conformer modules.

[0031] The lightweight Conformer module of the present invention is based on the original Conformer module, and uses dynamic convolution to replace part of the self-attention to reduce the computational complexity and model parameters. In the lightweight Conformer module, the input text feature channel is first masked into two parts to simplify the calculation, and then the local context information is extracted using dynamic convolution, and the global context information is extracted using the self-attention mechanism, and finally fused. Due to the masking operation on the feature channel, the amount of self-attention calculation here is also reduced, and this architecture can fuse local and global context information to improve the representation ability of the model. Figure 3 The improved Conformer module of the present invention, that is, the lightweight Conformer module adopted by the present invention, is shown.

[0032] like Figure 3 As shown in the figure, the lightweight Conformer is a module that integrates CNN and Transformer. It shows better accuracy with fewer parameters. At the same time, dynamic convolution and self-attention mechanism are used in conjunction. The basic idea of ​​dynamic convolution is to adaptively adjust the convolution parameters through attention according to the input data. The resulting convolution kernel weights are related to the input, that is, different data have different convolutions. Dynamic convolution can reduce the number of parameters, allow parallel calculation of different feature channels, and reduce linear computational complexity. In the lightweight Conformer module, self-attention learns global information, while convolution effectively captures local correlation. Finally, it fuses local and global context information to improve the representation ability of the model.

[0033] TTS is a one-to-many mapping problem. For a given text input, there may be multiple different speech outputs. The variation information will affect the expressiveness and naturalness of the synthesized speech. For Mongolian, which has a relatively small corpus, the expressiveness of the synthesized speech is more general.

[0034] The variable adapter is composed of a combination of multiple prediction modules, each of which is independent of each other. It is usually composed of a duration prediction module and a pitch prediction module. To solve the problem of insufficient speech prosody, the present invention adds a noise module and a speech feature prediction module to the variable adapter of the FastSpeech2 model. That is, the variable adapter of the present invention is composed of a duration prediction module, a pitch prediction module, a speech feature prediction module and a noise module. Figure 4As shown in the figure, the model structures of the duration prediction module, pitch prediction module and discourse feature prediction module are the same, but the model parameters are different. They are mainly composed of a 2-layer 1D convolutional network with ReLU activation, each of which is followed by LN and Dropout layers, as well as an additional linear layer.

[0035] The speech feature prediction module for Mongolian of the present invention extracts vectors from the speech data set used for training through an acoustic encoder, and then expands and adds them to the phoneme hidden sequence, wherein the phoneme hidden sequence is the output of the lightweight Conformer encoder. The noise module consists of a DPTNET model and a noise encoder. The DPTNET model is used to extract noise signals and is composed of an encoder, a decoder and a separation layer. The encoder and the decoder of the DPTNET model are both composed of one-dimensional convolutions. The encoder estimates the learning representation of the input signal, and the separation layer estimates the optimal mask to separate the noise in the signal. The decoder finally uses the mask provided by the separation layer to reconstruct the estimated signal source in the time domain. The noise encoder encodes the noise signal, and its structure is similar to the speech feature prediction model. The noise encoder consists of a 2-layer 1D convolutional network activated by ReLU, and each network is set after LN, Dropout and an additional linear layer for projecting the hidden state into the output sequence. The process of the entire noise module is: the DPTNET model extracts background noise audio, and then uses the noise encoder to convert it into noise conditions, and finally adds the noise conditions to the phoneme hidden sequence.

[0036] The overall schematic diagram of the acoustic model in the present invention is as follows Figure 5 As shown in the figure, the overall process of the model is as follows: the model inputs text, which is encoded by the encoder to provide text representation. The variable adapter embeds the aligned phoneme information through the sinusoidal position information to achieve hard alignment of the encoding step and the decoding step, and adds additional phoneme features. The decoder predicts and outputs a rough Mel spectrogram in parallel, and finally fine-tunes it through the post-processing network to obtain a refined Mel spectrogram.

[0037] The post-processing network of the present invention is a neural network composed of multiple one-dimensional convolutions, which predicts additional residual information through convolution to improve the generated Mel spectrum;

[0038] Step 4: Use the GAN-based lightweight model MelGAN as a vocoder to add Mongolian speech features. The Mel spectrogram is inferred and converted into a speech waveform to achieve Mongolian speech synthesis.

[0039] The present invention uses MelGAN acoustic features as conditional input to the vocoder to construct a MelGAN-based neural network vocoder. This model is a lightweight vocoder based on GAN, which can synthesize high-quality Mongolian speech at high speed on ordinary CPUs or GPUs. Figure 6 As shown, the vocoder of the present invention is composed of two sub-networks, the generator network and the discriminator network. The generator network is a combination of a one-dimensional transposed convolution and a local dilated attention block. The convolution extracts local information in the acoustic features, while the local dilated attention block expands the receptive field to obtain more information. The discriminator network is composed of a front and back one-dimensional convolution and four layers of grouped convolution. The discriminator network adjusts parameters through adversarial training so that the quality of the audio it generates is as close as possible to the quality of the real Mongolian audio.

[0040] The MelGAN vocoder of the present invention takes the Mel spectrum as input, gradually upsamples it to the speech length, adds dilated convolution to calculate the transformation from frequency domain to time domain between upsampling, and finally outputs Mongolian speech with a fixed number of frames.

[0041] In a specific embodiment of the present invention, in Mongolian: " is an example (its Chinese meaning is: health), its speech synthesis method is as follows:

[0042] 1. Extract text features, convert Mongolian text into the corresponding Latin characters "qihiragtv", and then use the character-to-phoneme model to obtain the corresponding phoneme representation: "qiragt vl"

[0043] 2. Pre-train the phonemes in the BERT pre-training model and add the corresponding syllables, morphemes and position information. Syllable information: "qi hi rag tv" morpheme "qihirag tv". The phoneme information containing position and speech features is obtained through pre-training.

[0044] 3. The phoneme information is fed into the acoustic model to generate the Mel spectrogram. The phoneme information is first extracted from the hidden features in the phoneme by the encoder and then processed by calculation to obtain the phoneme hidden sequence. Then the variable adapter adds additional speech feature information to the phoneme hidden sequence. The supplemented phoneme hidden sequence enters the decoder to synthesize the Mel spectrogram according to the feature representation in the phoneme hidden sequence. Finally, the synthesized coarse Mel spectrum is fine-tuned to obtain a refined Mel spectrogram.

[0045] 4. The refined Mel spectrum is sent to the vocoder, which generates a speech waveform through the Mel spectrum, and then adjusts the parameters through the identification network to optimize the synthesized speech. Finally, natural Mongolian speech is obtained.

Claims

1. A Mongolian speech synthesis method based on Conformer and MelGAN, characterized in that: The steps include: Step 1, obtaining Mongolian text data, extracting text features, converting the text feature encoding into a phoneme sequence, and inputting the phoneme sequence into a BERT pre-training model; the input of the BERT pre-training model consists of two segments, which are two different representations of the same content, the first segment is a phoneme sequence, including a single international standard phoneme as a first marker, and the second segment is a character sequence, including a single English character as a second marker; There are two special markers, CLS and SEP, in the input sequence. CLS is attached before the first segment and is used for additional classification targets besides the MLM target. SEP has two locations, attached after the first segment and after the second segment, respectively, to indicate segment boundaries. The syllable information and morpheme information of Mongolian are taken as input, position encoding is performed through the sine position function, and the association information of phonemes, syllables, characters and morphemes is constructed through the phoneme sequence and character sequence; Step 2: Use the BERT pre-training model to train Mongolian phoneme information, learn the rhythmic features in pronunciation, and align the phoneme information; Step 3: Using the FastSpeech2 acoustic model based on the lightweight Conformer module, acoustic features, namely the Mel spectrogram, are generated in parallel through phoneme information; The FastSpeech2 acoustic model includes an embedding layer, an encoder, a variable adapter, a decoder, and a post-processing network; the encoder and the decoder are both composed of a stack of lightweight Conformer modules; the model inputs text, which is encoded by the encoder to provide text representation, and the variable adapter embeds the alignment phoneme information through the sinusoidal position information to achieve hard alignment of the encoding step and the decoding step, and add additional phoneme features; The decoder predicts and outputs a rough Mel spectrum in parallel, and finally fine-tunes it through the post-processing network to obtain a refined Mel spectrum. The lightweight Conformer module is based on the original Conformer module and uses dynamic convolution to replace part of the self-attention. In the lightweight Conformer module, the input text feature channel is first shielded into two parts, and then the local context information is extracted by dynamic convolution, and the global context information is extracted by the self-attention mechanism, and finally fused. The post-processing network is a neural network composed of multiple one-dimensional convolutions, which predicts additional residual information through convolution to improve the generated Mel spectrum; The variable adapter is composed of a duration prediction module, a pitch prediction module, a speech feature prediction module and a noise module, wherein the duration prediction module, the pitch prediction module and the speech feature prediction module have the same model structure but different model parameters; the noise module is composed of a DPTNET model and a noise encoder; The speech feature prediction module extracts vectors from the speech dataset used for training through an acoustic encoder, and then expands and adds them to the phoneme hidden sequence, where the phoneme hidden sequence is the output of the lightweight Conformer encoder; The DPTNET model consists of an encoder, a decoder and a separation layer, wherein the encoder and the decoder are both composed of one-dimensional convolutions, the encoder estimates the learning representation of the input signal, the separation layer estimates the optimal mask to separate the noise in the signal, and the decoder finally uses the mask provided by the separation layer to reconstruct the estimated signal source in the time domain; The noise encoder consists of a 2-layer 1D convolutional network activated by ReLU. LN, Dropout and an additional linear layer are set after each network. The noise encoder is used to encode the noise signal; The process of the entire noise module is as follows: the DPTNET model extracts background noise audio, converts it into noise conditions using a noise encoder, and finally adds the noise conditions to the phoneme hidden sequence; Step 4: Use the GAN-based lightweight model MelGAN as a vocoder, add Mongolian speech features to it, and convert the Mel spectrogram into a speech waveform by reasoning, thereby realizing Mongolian speech synthesis.

2. The Mongolian speech synthesis method based on Conformer and MelGAN according to claim 1, characterized in that: In the step 1, the Mongolian text is first converted into a character code, then uniformly converted into Latin characters, and finally a phoneme sequence of the Mongolian text is obtained through a letter-to-phoneme model, where the phoneme sequence consists of a plurality of phonemes.

3. The Mongolian speech synthesis method based on Conformer and MelGAN according to claim 1, characterized in that: In step 2, the BERT pre-training model is an encoder composed of a series of stacked Conformer modules. The Conformer module calculates the dependency of related phonemes through a self-attention mechanism, making the correlation of phoneme features closer.

4. The Mongolian speech synthesis method based on Conformer and MelGAN according to claim 1, characterized in that: In step 4, the vocoder is composed of a generator network and a discriminator network. The generator network is a combination of a one-dimensional transposed convolution and a local dilated attention block. The convolution extracts local information in the acoustic features, and the local dilated attention block expands the receptive field. The discriminator network is composed of a front and back one-dimensional convolution and four layers of grouped convolution. The discriminator network adjusts parameters through adversarial training so that the quality of the audio generated by it is as close as possible to the quality of the real Mongolian audio.

5. The Mongolian speech synthesis method based on Conformer and MelGAN according to claim 1, characterized in that: The vocoder takes the Mel spectrum as input, gradually upsamples to the speech length, adds dilated convolution to calculate the transformation from frequency domain to time domain between upsampling, and finally outputs Mongolian speech with a fixed number of frames.

Images