Low-resource Lao speech synthesis method based on fine-grained prosody modeling

Abstract

The present invention relates to a low-resource Lao speech synthesis method based on fine-grained prosodic modeling, and belongs to the field of natural language processing. Due to the extreme scarcity of Lao speech resources, the traditional Tacotron2-based neural network speech synthesis method is difficult to fully train the model under extremely low-resource corpus conditions, resulting in the collapse of the model's attention mechanism and the lack of fluency and naturalness in the synthesized audio. The present invention proposes a low-resource Lao speech synthesis method with fine-grained prosodic modeling, which mainly includes two parts: preprocessing of Lao speech synthesis data and Lao speech synthesis based on fine-grained prosodic modeling. The method proposed in the present invention is to extract the tone features in Lao audio and the Lao character granularity pre-training language model, and fuse the granularity features of different levels through the cross-attention mechanism to help the model better learn the attention mechanism alignment relationship of the text-mel spectrogram, thereby improving the model's prosodic performance.

Description

Technical Field

[0001] The present invention relates to a low-resource Laotian speech synthesis method based on fine-grained prosody modeling, and belongs to the technical field of natural language processing. Background Art

[0002] Speech synthesis tasks usually require a large amount of text and audio data pairs to train the model. However, there are thousands of languages in the world, and resource-rich languages only account for a very small part of them. Most scarce languages lack training data. In the current end-to-end speech synthesis system, at least 20 hours of high-quality recordings of a single speaker are required. The size and breadth of the data have a great impact on the naturalness and intelligibility of the model. In low-resource situations, the Lao speech synthesis dataset is only a few hours or even an hour long. By directly modeling text to audio, it is difficult for the model to learn the attention mechanism alignment relationship from text to mel-spectrogram, resulting in the collapse of the attention mechanism, such as Figure 1 As shown, the synthesized audio will produce catastrophic errors such as completely irrelevant nonsense to the text. At the same time, the model is prone to overfitting with a small amount of data, the generalization ability of the model is reduced, and the synthesized audio is seriously lacking in naturalness and fluency.

[0003] Lao is a tonal language, which makes Lao speech synthesis more inclined to model tone symbols. The tones of Lao are: At the same time, Laotian uses tone symbols as separate characters after words. Different tones will result in different information and rhythmic expressions in the text. With a small amount of data, in order to improve the rhythmic expression, naturalness, and intelligibility of the Laotian speech synthesis model, it is necessary to incorporate the tone information in Laotian into the model training, which requires the attention mechanism to learn more knowledge. Summary of the Invention

[0004] The present invention provides a low-resource Lao speech synthesis method based on fine-grained prosodic modeling, proposes a Lao speech synthesis task under low resources, and solves the problems faced by Lao speech synthesis under low resources, such as the decline in model generalization ability and the lack of naturalness and fluency of synthesized audio.

[0005] The technical solution of the present invention is: a low-resource Lao speech synthesis method based on fine-grained prosodic modeling, the method comprising:

[0006] The specific steps of the method are as follows:

[0007] Step 1: Preprocess the Lao speech synthesis data, including data cleaning, encoding conversion, length ratio filtering, word segmentation, etc.

[0008] Step 2: Using the method of integrating external language knowledge, we integrate language knowledge of different granularities into the autoregressive model text encoder. We use the self-attention mechanism to integrate the character-granular Laotian pre-trained language model XLM-Roberta and the tacotron2 text encoder, and also use the self-attention mechanism to integrate the phoneme-granular audio prosodic feature f0 encoder and the tacotron2 text encoder. We use the tacotron2 decoder to predict the encoder's output hidden state, convert the hidden state into a mel-spectrogram feature, and finally generate the original waveform through the HiFiGAN model.

[0009] As a further solution of the present invention, the specific steps of Step 1 are:

[0010] Step 1.1. Lao text data cleaning: remove garbled characters in Lao text and standardize punctuation representation;

[0011] Step 1.2, Lao encoding conversion: Lao font unified encoding conversion into Unicode font;

[0012] Step 1.3, sentence length filtering: retain Lao sentences with a length of less than 250 characters to improve model training efficiency;

[0013] Step 1.4. Recording text audio: Laotians record 2,000 pre-processed Laotian text audio data with an audio sampling rate of 22.05 kHz. Each Laotian text and corresponding audio has a unique ID ranging from 1 to 2000.

[0014] Step 1.5. Build Mel-spectrogram: Use the conversion tool to convert the audio into Mel-spectrogram.

[0015] Step 1.6, Dictionary construction: Extract Lao characters from all Lao texts. After deduplication, the number of Lao characters is 73. Adding the unregistered word UNK, the constructed dictionary size is 74. Each character has an ID number.

[0016] At the data preprocessing level, Lao text data is cleaned, filtered by length ratio, and segmented. This is mainly to improve the quality of the speech synthesis model, standardize the speech synthesis process, and improve model performance.

[0017] As a further solution of the present invention, the specific steps of Step 2 are:

[0018] Step 2.1. First, convert the input text characters into the corresponding ID sequence through the dictionary, and then send the text ID sequence to the tacotron2 text encoder to generate the hidden state.

[0019] Step 2.2: The encoder integrates the text-level XLM-Roberta pre-trained language model and the phoneme-level prosodic feature f0 model. A cross-attention mechanism is used to focus on the deep relationships between multi-granular hidden states, enabling the model to better understand the contextual semantics of the input sequence. The outputs of the two cross-attention mechanisms are concatenated as the output of the model encoder.

[0020] As a further solution of the present invention, the specific steps of Step 2.2 are:

[0021] Step 2.2.1. Use the YAAPT algorithm to extract f0 from the input raw audio; and use the VQ-VAE framework to train a pre-trained model for the Lao audio pitch feature f0; in order to generate a discrete f0 representation, use the VQ-VAE encoder Generate unquantized f0 features from the text, and then use The unquantized f0 is quantized; the encoder extracts a latent vector sequence from the original audio, and then the encoder maps each latent vector to the nearest vector in the codebook, and then feeds the embedded latent vector to the decoder to reconstruct the original f0 signal; the encoder obtained by training It is incorporated into the acoustic model encoder as external knowledge with Laotian pitch information.

[0022] pass and Generate quantized f0 features from a piece of text, and use Tacotron2 encoder (Encoder tacotron2 ) is represented as The initial text is trained through the XLM-roberta pre-trained language model (Encoder XLM-roberta ) is converted into text-granular hidden state features. Then, the output hidden states of the three encoders are fused through the cross-attention mechanism, and the output of the cross-attention is concatenated and fed into the decoder;

[0023] The following formula is used to express the formula flow of Lao speech synthesis integrating multi-granularity prosodic modeling:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] Among them, x 1:n is a text sequence with an input length of n, φ F is a vector quantization function. Cross_attention(,) is to extract the features from one hidden state to another hidden state and fuse the features of the latter to the former. Here, the output of the f0 encoder is and the output of the XLM-roberta model Fused to the output of the tacotron2 text encoder It is the hidden state of the text encoder and the pitch feature f0 encoder. It is a text hidden state that integrates the XLM-roberta pre-trained language model. It concatenates two output hidden states through cat() and then sends them to the decoder Decoder() to realize acoustic feature prediction. The result is mel pre ;

[0032] Step 2.2.2: The output of the encoder passes through a double-layer LSTM model and multiple linear layers in the decoder and is finally converted into a coarse-grained Mel-spectrogram. The coarse-grained Mel-spectrogram is output as a fine-grained Mel-spectrogram through the postnet module The final result is output as a Mel spectrum graph with two granularities The loss function of the model of the present invention is defined as:

[0033]

[0034] Where l is the maximum value between the real Mel spectrum frame length and the predicted Mel spectrum frame length, is the predicted value of the i-th frame of the coarse-grained Mel-spectrogram, is the predicted value of the i-th frame of the fine-grained Mel-spectrogram, loss g is the introduced guided attention loss, and α is the weight.

[0035] Finally, after training a suitable Lao speech synthesis model, the mel spectrogram is converted into the original waveform through the hifigan vocoder.

[0036] The beneficial effects of the present invention are: the present invention is capable of performing speech synthesis tasks under low resources; the present invention enriches the latent representation of the encoder output and improves the generalization ability of the model under a small amount of data; the present invention can better reproduce the tone characteristics of the original audio; the present invention has a good effect on modeling the tone characteristics of Laotian speech; the present invention extracts the tone characteristics in Laotian audio and the Laotian character granularity pre-trained language model, and fuses the granularity features of different levels through the cross-attention mechanism, helping the model to better learn the attention mechanism alignment relationship between text and mel-spectrogram, thereby improving the rhythmic performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the low-resource Lao speech synthesis method based on fine-grained prosody modeling in the present invention;

[0038] Figure 2 A flowchart of the method of the present invention;

[0039] Figure 3 This is a schematic diagram showing an example comparison of the pitch features of the Lao language sequence synthesized audio and real audio in the present invention. DETAILED DESCRIPTION

[0040] Example 1: Figure 1-Figure 3 As shown, a low-resource Lao speech synthesis method based on fine-grained prosody modeling comprises:

[0041] Step 1: Preprocess the Lao speech synthesis data, including data cleaning, encoding conversion, length ratio filtering, and word segmentation. At the data preprocessing level, data cleaning, length ratio filtering, and word segmentation are performed on the Lao text data to improve the quality of the speech synthesis model, standardize the speech synthesis process, and improve model performance.

[0042] The specific steps of Step 1 are:

[0043] Step 1.1. Lao text data cleaning: remove garbled characters in Lao text and standardize punctuation representation;

[0044] Step 1.2, Lao encoding conversion: Lao font unified encoding conversion into Unicode font;

[0045] Step 1.3, sentence length filtering: retain Lao sentences with a length of less than 250 characters to improve model training efficiency.

[0046] Step 1.4. Record text audio: Laotians recorded two thousand pre-processed Laotian text audio data with an audio sampling rate of 22.05 kHz. Each Laotian text and corresponding audio has a unique ID ranging from 1 to 2000.

[0047] Step 1.5. Build Mel-spectrogram: Use the conversion tool to convert the audio into Mel-spectrogram.

[0048] Step 1.6, Dictionary construction: Extract Lao characters from all Lao texts. After deduplication, the number of Lao characters is 73. Adding the unregistered word UNK, the constructed dictionary size is 74. Each character has an ID number.

[0049] Step 2. Lao speech synthesis method based on fine-grained prosody modeling: The method of integrating external language knowledge is adopted to integrate language knowledge of different granularities into the autoregressive model text encoder. The character-granular Lao pre-trained language model XLM-Roberta and the tacotron2 text encoder are integrated through the self-attention mechanism, and the audio prosody feature f0 encoder and the tacotron2 text encoder at the phoneme granularity are integrated through the self-attention mechanism; the tacotron2 decoder is used to predict the output hidden state of the encoder, and the hidden state is converted into an 80-dimensional Mel-spectrogram feature. Finally, the original waveform is generated through the hifigan model.

[0050] The specific steps of Step 2 are:

[0051] Step 2.1. First, convert the input text characters into the corresponding ID sequence through the dictionary, and then send the text ID sequence to the tacotron2 text encoder to generate the hidden state.

[0052] Step 2.2: The encoder integrates the text-level XLM-Roberta pre-trained language model and the phoneme-level prosodic feature f0 model. A cross-attention mechanism is used to focus on the deep relationships between multi-granular hidden states, enabling the model to better understand the contextual semantics of the input sequence. The outputs of the two cross-attention mechanisms are concatenated as the output of the model encoder.

[0053] The specific steps of Step 2.2 are:

[0054] Step 2.2.1. Use the YAAPT algorithm to extract f0 from the input raw audio. And use the VQ-VAE framework to train a pre-trained model for the Lao audio pitch feature f0. In order to generate a discrete f0 representation, the VQ-VAE encoder is used. Generate unquantized f0 features from the text, and then use The unquantized f0 is quantized. The encoder extracts a sequence of latent vectors from the original audio. The encoder then maps each latent vector to the nearest vector in the codebook, feeds the embedded latent vector to the decoder, and reconstructs the original f0 signal. The encoder obtained by training It is incorporated into the acoustic model encoder as external knowledge with Laotian pitch information;

[0055] pass and Generate quantized f0 features from a piece of text, and use Tacotron2 encoder (Encoder tacotron2 ) is represented as The initial text is trained through the XLM-roberta pre-trained language model (Encoder XLM-roberta ) is converted into text-granular hidden state features. Then, the output hidden states of the three encoders are fused through the cross-attention mechanism, and the output of the cross-attention is concatenated and fed into the decoder.

[0056] The following formula is used to express the formula flow of Lao speech synthesis integrating multi-granularity prosodic modeling:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] Among them, x 1:n is a text sequence with an input length of n, φF is a vector quantization function, Cross_attention(,) is to extract the features from one hidden state to another hidden state, and fuse the features of the latter to the former. Here, the output of the f0 encoder is and the output of the XLM-roberta model Fused to the output of the tacotron2 text encoder It is the hidden state of the text encoder and the pitch feature f0 encoder. It is a text hidden state that integrates the XLM-roberta pre-trained language model. It concatenates two output hidden states through cat() and then sends them to the decoder Decoder() to realize acoustic feature prediction. The result is mel pre .

[0065] Step 2.2.2: The output of the encoder passes through a double-layer LSTM model and multiple linear layers in the decoder and is finally converted into a coarse-grained Mel-spectrogram. The coarse-grained Mel-spectrogram is output as a fine-grained Mel-spectrogram through the postnet module The final result is output as a Mel spectrum graph with two granularities The loss function of the model of the present invention is defined as:

[0066]

[0067] Where l is the maximum value between the real Mel spectrum frame length and the predicted Mel spectrum frame length, is the predicted value of the coarse-grained Mel spectrum frame i, is the predicted value of the i-th frame of the fine-grained Mel-spectrogram, loss g is the introduced guided attention loss, and α is the weight.

[0068] Finally, after training a suitable Lao speech synthesis model, the mel spectrogram is converted into the original waveform through the hifigan vocoder.

[0069] To illustrate the effectiveness of the present invention, the following experiments were conducted: Experiments were conducted on the public English dataset LJSpeech and an internal Laotian dataset, using 22.05kHz audio as input and 80-dimensional mel-spectrograms as intermediate output. First, in the English dataset experiment, to compare the impact of corpus size on the model, the present invention trained the model on 10k, 5k, 2k, and 1k English datasets, with a validation set size of 0.5k. The 10k speech dataset totaled approximately 20 hours. Next, in the Laotian dataset experiment, the Laotian text was crawled from CRI and manually screened to select a high-quality 2.5k plain text corpus. The audio for the 2.5k plain text was recorded by a native Laotian speaker in a professional recording studio, totaling approximately 2 hours. The training and validation sets had a 4:1 size ratio, with 2k and 0.5k, respectively. This paper adds a cross-attention mechanism and guided attention loss to the Tacotron2 model as the baseline model of the present invention. Use parameters β1 = 0.9, β2 = 0.999, ε = 10 -6 , Adam optimizer, learning rate is 10 -3, set the Guide Attention Loss weight α = 1.

[0070] In order to verify the effectiveness of the inventive method proposed in the present invention, the following comparative experiments were designed for analysis. Three groups of experiments were conducted on the dataset.

[0071] Experiment 1: Comparison of the MOS scores of the tacotron2 model and XLM-f0-Tac at four corpus sizes. Model 1: tacotron2 model. Four English corpus sizes were used. Specific corpus size-based experimental results are shown in Table 1. In the corpus size column, the first column indicates the total number of corpus items, and the second column indicates the total audio duration. Model 2: XLM-f0-Tac. This experiment was conducted on the smallest corpus size, combining the XLM-roberta pre-trained language model and a Laotian audio prosody and intonation model trained using VQ-VAE.

[0072] Table 1: MOS scores for four different English corpus sizes using baseline models. Tac represents the Tacotron2 architecture, while XLM-F0-Tac integrates a multi-granularity prosodic feature model based on a cross-headed multi-head attention mechanism. The latter two methods were tested using the addition of a guided attention loss.

[0073]

[0074] When the corpus size reaches 10k, a total of about 20 hours of audio, the overall MOS score is the highest, reaching 4.52 points. As the corpus size continues to decrease, the naturalness and intelligibility of the speech will gradually decrease. As the corpus size decreases to 2 hours, the speech synthesis model based on the tacotron2 architecture cannot train a usable system. This is because the scale of the training data decreases as the data size decreases, the less knowledge the model learns, and the generalization ability of the model decreases, resulting in problems with the naturalness and fluency of the synthesized audio. When the training data decreases to the minimum data size required by the model, the model's attention collapses, and the model's attention mechanism cannot learn complete text-to-audio knowledge, resulting in a decrease in the fluency and naturalness of the model's synthesized audio. The present invention expands the latent representation information output by the encoder, so that the model can still train a usable speech synthesis system under the condition of a corpus size of 1 hour, and obtains a MOS value score of 3.93. Experiments have shown that the model of the present invention is capable of speech synthesis tasks under low resources.

[0075] Experiment 2: Laotian speech synthesis evaluation and ablation experiment. The feasibility of the model was verified on an English dataset of approximately one hour. The Laotian model was trained using 2k of text and approximately one hour of aligned audio. Table 2 compares the MOS scores of the Laotian speech synthesis audio and real audio.

[0076] Table 2: MOS scores of Laotian synthesis under two model architectures for a 2-hour corpus under the baseline model. Truth_mel is the audio output of the true Mel spectrogram through the vocoder.

[0077]

[0078] The table above demonstrates that the model presented in this paper can train a complete speech synthesis model using only one hour of Laotian audio text training data. By fine-grained modeling of Laotian speech tonal characteristics, the f0-based pre-training model can significantly improve the prosodic performance of the Laotian speech synthesis model, even with limited resources, and achieve a higher score. This demonstrates that the present invention can demonstrate impressive performance even with a 2k corpus.

[0079] To further validate the advantages of this invention, we conducted ablation experiments on a baseline model to examine the impact of the pre-trained language model and Laotian tone model on speech synthesis. We tried various approaches to improve model performance, but most yielded poor results, with attention maps failing to align. We selected a model architecture that performed well for demonstration purposes. Table 3 shows the results of the ablation experiments.

[0080] Table 3: Laotian MOS scores under different models. Guide attention loss is abbreviated as GA loss. XLM-Tac, F0-Tac, and XLM-F0-Tac integrate guide attention loss by default.

[0081]

[0082] The acoustic model without the XLM-Roberta pre-trained language model obtained the lowest MOS score. After adding the XLM-Roberta pre-trained language model, the naturalness of the synthesized audio was improved. This is because the model of the present invention enriches the implicit representation of the encoder output and improves the generalization ability of the model with a small amount of data. At the same time, the acoustic model with the addition of GuideAttention Loss can also train a complete speech synthesis system, because this method is conducive to strengthening text-audio alignment and accelerating training. Modeling Laotian intonation effectively improves the rhythmic performance of the synthesized audio. The baseline model that added the pre-trained language model, tone feature model and loss method at the same time obtained the highest MOS score. Through this ablation experiment, the importance of integrating fine-grained rhythmic features in the model of the present invention was proved.

[0083] In order to more intuitively demonstrate the modeling ability of the model on the Laotian tone characteristics, the present invention Speech synthesis is performed, and f0 is calculated on the spectrum through the algorithm, such as Figure 3 As shown, Figure 3 Lao sequences The tonal characteristics of the synthesized audio and the real audio are compared in the examples. Figure 3 The upper and lower figures are the representations of the pitch features of real audio and synthesized audio on the spectrum graph, respectively.

[0084] By comparing the pitch features of real audio and synthesized audio on the spectrum, it can be clearly seen that the present invention can better reproduce the pitch features of the original audio. This is because the pitch feature information is incorporated into the model training process, and the pitch feature information can better reflect the rhythmic characteristics of the audio. The pitch features are mapped to each character through the attention mechanism, making the encoder's output hidden state richer in deeper information. Experimental results also show that the present invention is effective in modeling the pitch features of Laotian speech.

[0085] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A low-resource Lao speech synthesis method based on fine-grained prosodic modeling, characterized by: The specific steps of the method are as follows: Step 1: Preprocess the Lao speech synthesis data, including data cleaning, encoding conversion, length ratio filtering, and word segmentation. Step 2: Using the method of integrating external language knowledge, the language knowledge of different granularities is integrated into the autoregressive model text encoder. The Tacotron2 encoder integrates the XLM-roberta pre-trained language model of text granularity and the rhythmic feature f0 model of phoneme granularity. tacotron2 The output hidden state and initial text are passed through the XLM-roberta pre-trained language model Encoder XLM-roberta The hidden state features converted into text granularity are fused through the cross attention mechanism Cross_attention, Tacotron2 encoder Encoder tacotron2 The output hidden state is fused with the audio prosody feature f0 at the phoneme granularity through the cross-attention mechanism. The outputs of the two cross-attention mechanisms are spliced as the output of the model encoder. The tacotron2 decoder is used to predict the encoder's output hidden state, convert the hidden state into a mel-spectrogram feature, and finally generate the original waveform through the hifigan model.

2. The low-resource Lao speech synthesis method based on fine-grained prosodic modeling according to claim 1, characterized in that: The specific steps of Step 1 are: Step 1.

1. Lao text data cleaning: remove garbled characters in Lao text and standardize punctuation representation; Step 1.2, Lao encoding conversion: Lao font unified encoding conversion into Unicode font; Step 1.3, sentence length filtering: retain Lao sentences with a length of less than 250 characters to improve model training efficiency; Step 1.

4. Recording text audio: Laotians record 2,000 pre-processed Laotian text audio data with an audio sampling rate of 22.05 kHz. Each Laotian text and corresponding audio has a unique ID ranging from 1 to 2000. Step 1.

5. Build Mel-spectrogram: Use the conversion tool to convert the audio into Mel-spectrogram. Step 1.6, Dictionary construction: Extract Lao characters from all Lao texts. After deduplication, the number of Lao characters is 73. Adding the unregistered word UNK, the constructed dictionary size is 74. Each character has an ID number.

3. The low-resource Lao speech synthesis method based on fine-grained prosodic modeling according to claim 1, characterized in that: The specific steps of Step 2 are: Step 2.1: First, convert the input text characters into corresponding ID sequences through the dictionary, and then feed the text ID sequence into the tacotron2 text encoder to generate hidden states; Step 2.2: The encoder integrates the text-granular XLM-roberta pre-trained language model and the phoneme-granular rhythmic feature f0 model, and uses the cross-attention mechanism to focus on the deep relationship between multi-granularity hidden states, so that the model can better understand the contextual semantic information of the input sequence. The outputs of the two cross-attention mechanisms are spliced as the output of the model encoder.

4. The low-resource Lao speech synthesis method based on fine-grained prosodic modeling according to claim 3 is characterized in that: The specific steps of Step 2.2 are: Step 2.2.

1. Use the YAAPT algorithm to extract f0 from the input raw audio; and use the VQ-VAE framework to train a pre-trained model for the Lao audio pitch feature f0; in order to generate a discrete f0 representation, use the VQ-VAE encoder Generate unquantized f0 features from the text, and then use The unquantized f0 is quantized; the encoder extracts a sequence of latent vectors from the original audio, then the encoder maps each latent vector to the nearest vector in the codebook, feeds the embedded latent vector to the decoder, and reconstructs the original f0 signal; The trained encoder It is incorporated into the acoustic model encoder as external knowledge with Laotian pitch information; pass and Generate quantized f0 features from a piece of text, and use Tacotron2 encoder tacotron2 The hidden state is represented as The initial text is passed through the XLM-roberta pre-trained language model Encoder XLM-roberta Converted into text-granular hidden state features; Then the output hidden states of the three encoders are fused through the cross attention mechanism, and the output of the cross attention is concatenated and sent to the decoder; The following formula is used to express the formula flow of Lao speech synthesis integrating multi-granularity prosodic modeling: Among them, x 1:n is a text sequence with an input length of n, φ F is a vector quantization function. Cross_attention(,) is to extract the features from one hidden state to another hidden state and fuse the features of the latter to the former. Here, the output of the f0 encoder is and the output of the XLM-roberta model Fused to the output of the tacotron2 text encoder It is the hidden state of the text encoder and the pitch feature f0 encoder. It is a text hidden state that integrates the XLM-roberta pre-trained language model. It concatenates two output hidden states through cat() and then sends them to the decoder Decoder() to realize acoustic feature prediction. The result is mel pre ; Step 2.2.2: The output of the encoder passes through a double-layer LSTM model and multiple linear layers in the decoder and is finally converted into a coarse-grained Mel-spectrogram. The coarse-grained Mel-spectrogram is output as a fine-grained Mel-spectrogram through the postnet module The final result is output as a Mel spectrum graph with two granularities The loss function of the model is defined as: Where l is the maximum value between the real Mel spectrum frame length and the predicted Mel spectrum frame length, is the predicted value of the i-th frame of the coarse-grained Mel-spectrogram, is the predicted value of the i-th frame of the fine-grained Mel-spectrogram, loss g is the introduced guided attention loss, α is the weight; Finally, after the Lao speech synthesis model is trained, the mel spectrogram is converted into the original waveform through the hifigan vocoder.

Images