Phoneme knowledge enhanced old-English mixed language speech synthesis method and device

Abstract

The invention relates to a phoneme knowledge enhanced old-English mixed language speech synthesis method and device, and belongs to the technical field of natural language processing. The method comprises the following steps: constructing a phoneme fine-grained pronunciation knowledge dictionary; a phoneme knowledge enhanced old-English mixed language speech synthesis model is constructed, and the phoneme knowledge enhanced old-English mixed language speech synthesis model comprises the steps that fine-grained pronunciation features are fused in the encoding stage, and pronunciation is directly modeled; during model training, firstly, characters are converted into id identifiers through a phoneme fine-grained pronunciation knowledge dictionary, and then vector representation is generated through an embedding layer, so that traditional phoneme embedding is obtained; then, representing the pronunciation characteristics of the phonemes as a one-dimensional matrix, and distinguishing the pronunciation differences of the phonemes by setting a value in the matrix as 1 or-1, thereby modeling fine-grained pronunciation characteristics; and old-English mixed language speech synthesis is carried out. According to the invention, pronunciation synthesis can be carried out more accurately.

Description

Technical Field

[0001] The invention relates to a method and device for synthesizing an Old-English mixed language speech with enhanced phoneme knowledge, and belongs to the technical field of natural language processing. Background Art

[0002] With the development of artificial intelligence technology, the effects of acoustic models and vocoder models for speech synthesis are constantly improving. When the data scale is sufficient, the speech synthesis task of a single language can already synthesize high-quality speech. However, as the demand develops with technology, mixed-language speech synthesis is gradually being valued. Mixed-language speech synthesis has become relatively mature in recent years, and has achieved excellent results in mixed-language synthesis tasks on resource-rich languages. However, since English words occasionally appear in Lao, when speech synthesis is required for such texts, it is found that Lao people cannot accurately pronounce English. Because the language representation in mixed languages ​​is different, it is difficult for Lao people to pronounce across English phonemes.

[0003] Each language has its own unique phoneme set, which makes it difficult to achieve effective pronunciation of mixed-language phonemes for speakers. In practice, the unique phoneme sets of Lao and English make it very difficult to transfer speaker phonemes when synthesizing audio, reducing the fluency and naturalness of the entire model. For example, the phonemes in Lao are " h " is a phoneme that does not exist in English. Therefore, when synthesizing Lao audio using English speakers, it is difficult to recognize the unseen phonemes" h "Achieve pronunciation. Therefore, in the mixed language speech synthesis task, how to achieve a unified representation of mixed languages ​​from text to pronunciation is the key. Existing methods usually take text sequences or phoneme sequences as input. In the process of text or phoneme to sound, the model will explicitly ignore the hidden features of phonemes, and purely realize the synthesis of phonemes into fixed sounds. If the hidden information of phonemes can be explicitly used as one of the inputs in the input sequence, then when the model realizes speaker mixed language synthesis, especially for invisible phonemes in speaker transfer, its explicit high-level features can be modeled at a deeper level, thereby improving the naturalness and fluency of the synthesized audio when transferring speakers. Summary of the invention

[0004] The present invention provides a mixed-language speech synthesis method and device for Lao-English enhanced with phoneme knowledge, which helps Lao speakers to effectively pronounce English phonemes when the model is inferring by adding speaker-independent phoneme fine-grained pronunciation features as explicit pronunciation features and fusing the pronunciation features with encoder features.

[0005] The technical solution of the present invention is: a method for speech synthesis of an Old-English mixed language enhanced by phoneme knowledge, the method comprising:

[0006] Step 1: Construct a phoneme fine-grained pronunciation knowledge dictionary;

[0007] Step 2: Construct an English-Chinese mixed language speech synthesis model enhanced with phoneme knowledge. The English-Chinese mixed language speech synthesis model enhanced with phoneme knowledge includes:

[0008] Fine-grained pronunciation features are incorporated in the encoding stage to directly model pronunciation;

[0009] When training the model, the characters are first converted into IDs through the phoneme fine-grained pronunciation knowledge dictionary, and then a vector representation is generated through the embedding layer to obtain the traditional phoneme embedding. Then, the pronunciation features of the phonemes are represented as a one-dimensional matrix, and the differences in phoneme pronunciations are distinguished by setting the values ​​in the matrix to 1 or -1, thereby modeling the fine-grained pronunciation features. Combining the phoneme pronunciation feature vector with the traditional phoneme embedding sequence, the model can accurately model the phoneme pronunciations of Lao and English during the training phase, which is used to alleviate the difficulty of unified modeling of mixed language phonemes.

[0010] Step 3, Lao-English mixed language speech synthesis: The Lao-English mixed language speech synthesis model enhanced with phoneme knowledge in step 2 is integrated into an API interface that can be called on the server. By calling the model API interface, a Lao-English mixed language speech synthesis device is constructed for Lao-English mixed language speech synthesis.

[0011] Furthermore, in step 1, the IPA International Phonetic Alphabet is used to uniformly represent Lao and English as phonemes, and a phoneme fine-grained pronunciation knowledge dictionary is constructed on all phoneme sequences.

[0012] Furthermore, the specific steps of step 1 include:

[0013] Step 1.1 Phoneme dictionary construction: The Lao data crawled from the Internet were manually annotated, including the removal of useless characters; then, the Lao and English texts were converted into phoneme sequences using Epitran and Espeak tools, respectively, to preliminarily construct a phoneme dictionary;

[0014] Step 1.2 Processing of phoneme fine-grained pronunciation knowledge: Based on the phoneme dictionary constructed in step 1.1, symbols and attribute tags are added so that each character has additional attributes, and the pronunciation of different characters is marked by different attribute values.

[0015] Furthermore, in step 2, the operation process of the Old-English mixed language speech synthesis model enhanced with phoneme knowledge includes:

[0016] Step 2.1: Based on the phoneme fine-grained pronunciation knowledge dictionary constructed in step 1, a one-hot encoding method is used to construct a one-dimensional all-zero matrix of equal length according to the size of the attribute dictionary. According to the attribute value of each phoneme marked in the fine-grained pronunciation dictionary, the corresponding label position in the one-dimensional all-zero matrix is ​​set to 1, and the opposite type is set to -1 to increase the gap between phonemes with far different pronunciations, thereby embedding the phoneme fine-grained pronunciation attributes, and then introducing the LSTM network to output the hidden state of the phoneme sequence on the fine-grained pronunciation embedding; the size of the input fine-grained pronunciation feature vector is:

[0017] shape(Input vector ) = B*L*H1;

[0018] Among them, Input vector It is represented as the phoneme fine-grained pronunciation feature, B is the batch size, L is the longest sentence length in a batch, and H1 represents the embedding dimension of each character;

[0019] The phoneme fine-grained pronunciation feature vector is passed through the LSTM network to obtain the fine-grained pronunciation hidden state h lstm , the Cain state is expressed as:

[0020] h lstm =LSTM(Input vector );

[0021] Step 2.2: Use a text convolutional encoder to directly convert characters into IDs according to the pronunciation dictionary, and then obtain the phoneme embedding h through embedding. conv , the embedding dimension is 256;

[0022] Step 2.3: Based on the hidden state values ​​of the two phoneme granularities obtained in Step 2.1 and Step 2.2, the fine-grained pronunciation hidden state h is converted into lstm Fused into the output vector h of the text encoder conv In order to increase the feature information of the encoder output; finally, the output of the encoder is further processed by the residual network structure, in which the fine-grained pronunciation hidden state h lstm As the cross-attention k and v, the output vector h of the text encoder conv As q, it is calculated as follows:

[0023] h att = Cross_att(h conv ,h lstm ,h lstm )

[0024] h att represents the output of cross attention;

[0025] The attention output is then added to the encoder and fed into layer normalization:

[0026] h l_n =nn.layernorm(h att +dropout(h conv ))

[0027] After the normalization layer, a linear layer is first connected, then a ReLU activation function is passed, and then another linear layer; finally, the output of the linear layer is compared with h l_n Add together to get the final output representation of the encoder h out , calculated as follows:

[0028] h out =nn.laynorm(fc(max(0,fc(h l_n )))+dropout(h l_n ))

[0029] fc represents the linear layer, dropout (h l_n ) represents the output h of the normalization layer l_n Apply the Dropout operation to reduce the model's dependence on the output features of the normalization layer;

[0030] Step 2.4: Output h obtained by Step 2.3 out At the same time, it is sent to the decoder and speaker classifier module, and the decoder is used to predict the speech; in the speaker classifier module, h out First, it passes through the gradient inversion module to ensure that the output of the gradient inversion module remains unchanged during forward propagation, then passes through two linear layers, and finally uses the Softmax function to achieve speaker classification prediction.

[0031] Furthermore, the specific steps of step 3 are:

[0032] Step 3.1, deploy the trained ".pt" format model to the server, and implement the function of multi-user concurrent requests by calling the model, and build a Lao-English mixed language speech synthesis device, which includes a client and a server;

[0033] Step 3.2: The user inputs the original Lao text to be synthesized into the synthesis device client, and the client passes the input text to the translation model on the server; the model is responsible for synthesizing the input text into the target speech and returning the synthesized speech to the client interface for playback.

[0034] The device for speech synthesis of Old and English mixed languages ​​enhanced by phoneme knowledge comprises: a module for executing the method for speech synthesis of Old and English mixed languages ​​enhanced by phoneme knowledge.

[0035] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the Old-English mixed language speech synthesis method with enhanced phoneme knowledge is implemented.

[0036] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the Old-English mixed language speech synthesis method enhanced with phoneme knowledge.

[0037] A computer program product comprises a computer program, which, when executed by a processor, implements the Old-English mixed language speech synthesis method enhanced with phoneme knowledge.

[0038] The beneficial effects of the present invention are:

[0039] The present invention introduces pronunciation feature information that is invisible in phoneme embedding. Each phoneme and symbol is marked with a type and multiple pronunciation attributes. The model can see more information about the phoneme during the training stage, so that pronunciation synthesis can be performed more accurately when converting phonemes to audio, effectively alleviating the difficulty of unified modeling of mixed language phonemes and speaker timbre. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the present invention;

[0041] Figure 2 This is a network architecture diagram of the Old-English mixed language speech synthesis with enhanced phoneme knowledge of the present invention;

[0042] Figure 3 This is a schematic diagram of the subjective MOS evaluation score of the ablation experiment in the present invention;

[0043] Figure 4 Schematic diagram of the changes in the Mel-spectrogram of PP-Tacotron2 in the present invention and the benchmark model when Lao people speak English. DETAILED DESCRIPTION

[0044] Example 1: Figure 1-Figure 4 As shown, the phoneme knowledge enhanced Old-English mixed language speech synthesis method and device, the method comprises:

[0045] Step 1: Construct a phoneme fine-grained pronunciation knowledge dictionary;

[0046] Step 2: Construct an English-Chinese mixed language speech synthesis model enhanced with phoneme knowledge. The English-Chinese mixed language speech synthesis model enhanced with phoneme knowledge includes:

[0047] Fine-grained pronunciation features are incorporated in the encoding stage to directly model pronunciation;

[0048] When training the model, the characters are first converted into IDs through the phoneme fine-grained pronunciation knowledge dictionary, and then a vector representation is generated through the embedding layer to obtain the traditional phoneme embedding. Then, the pronunciation features of the phonemes are represented as a one-dimensional matrix, and the differences in phoneme pronunciations are distinguished by setting the values ​​in the matrix to 1 or -1, thereby modeling the fine-grained pronunciation features. Combining the phoneme pronunciation feature vector with the traditional phoneme embedding sequence, the model can accurately model the phoneme pronunciations of Lao and English during the training phase, which is used to alleviate the difficulty of unified modeling of mixed language phonemes.

[0049] Step 3, Lao-English mixed language speech synthesis: The Lao-English mixed language speech synthesis model enhanced with phoneme knowledge in step 2 is integrated into an API interface that can be called on the server. By calling the model API interface, a Lao-English mixed language speech synthesis device is constructed for Lao-English mixed language speech synthesis.

[0050] Furthermore, in step 1, the IPA International Phonetic Alphabet is used to uniformly represent Lao and English as phonemes, and a phoneme fine-grained pronunciation knowledge dictionary is constructed on all phoneme sequences.

[0051] Furthermore, the specific steps of step 1 include:

[0052] Step 1.1 Phoneme dictionary construction: The Lao data crawled from the Internet were manually annotated, including the removal of useless characters; then, the Lao and English texts were converted into phoneme sequences using Epitran and Espeak tools, respectively, to preliminarily construct a phoneme dictionary;

[0053] Step 1.2 Processing of phoneme fine-grained pronunciation knowledge: Based on the phoneme dictionary constructed in step 1.1, add symbols and attribute tags so that each character has additional attributes, and mark the pronunciation of different characters by assigning different attribute values. For specific attributes, please see Table 1. symbol_type indicates the type of each character. If it is a phoneme, it is marked as ph, and if it is a symbol, it is marked as other attribute values, such as '? ':{'symbol_type':'qm'}, '! ':{'symbol_type':'em'}. The fixed attributes of each phoneme are vowel_consonant and VUV attributes. The vowel_consonant attribute divides the phonemes into three types: vowels, consonants, and others. VUV marks whether the pronunciation type is unvoiced or voiced. Different major categories of phonemes will affect other phoneme attributes.

[0054] For example, the consonant '∫' symbol in the phoneme sequence converted from the English dataset:

[0055]

[0056]

[0057] Individual phoneme symbols symbol_type are marked as ph type, and additional phoneme attributes are added, whether it is a vowel consonant vowel_consonant, unvoiced or voiced VUV, and for consonants, there are two attributes consonant_place and consonant_manner.

[0058] For vowels, symbol:

[0059]

[0060] In addition to the symbol_type, vowel_consonant, and VUV attributes, each vowel also has vowel-specific attributes vowel_frontness, vowel_openness, and vowel_roundedness, which are fine-grained representations of the vowel in pronunciation, position, and roundness, respectively.

[0061] The present invention uses the above method to perform more fine-grained feature representation on all phonemes and symbols in English and Lao, thereby constructing an explicit phoneme fine-grained pronunciation knowledge dictionary.

[0062] Table 1 Phoneme fine-grained pronunciation attributes and attribute values

[0063]

[0064] Furthermore, in step 2, the operation process of the Old-English mixed language speech synthesis model enhanced with phoneme knowledge includes:

[0065] Step 2.1: Based on the phoneme fine-grained pronunciation knowledge dictionary constructed in step 1, a one-hot encoding method is used to construct a one-dimensional all-zero matrix of equal length according to the size of the attribute dictionary. According to the attribute value of each phoneme marked in the fine-grained pronunciation dictionary, the corresponding label position in the one-dimensional all-zero matrix is ​​set to 1, and the opposite type is set to -1 to increase the gap between phonemes with far different pronunciations, thereby embedding the phoneme fine-grained pronunciation attributes, and then introducing the LSTM network to output the hidden state of the phoneme sequence on the fine-grained pronunciation embedding; the size of the input fine-grained pronunciation feature vector is:

[0066] shape(Input vector ) = B*L*H1;

[0067] Among them, Input vectorIt is represented as the phoneme fine-grained pronunciation feature, B is the batch size, L is the longest sentence length in a batch, and H1 represents the embedding dimension of each character;

[0068] The phoneme fine-grained pronunciation feature vector is passed through the LSTM network to obtain the fine-grained pronunciation hidden state h lstm , the Cain state is expressed as:

[0069] h lstm =LSTM(Input vector );

[0070] Step 2.2: Use a text convolutional encoder to directly convert characters into IDs according to the pronunciation dictionary, and then obtain the phoneme embedding h through embedding. conv , the embedding dimension is 256;

[0071] Step 2.3: Based on the hidden state values ​​of the two phoneme granularities obtained in Step 2.1 and Step 2.2, the fine-grained pronunciation hidden state h is converted into lstm Fused into the output vector h of the text encoder conv In order to increase the feature information of the encoder output; finally, the output of the encoder is further processed by the residual network structure, in which the fine-grained pronunciation hidden state h lstm As the cross-attention k and v, the output vector h of the text encoder conv As q, it is calculated as follows:

[0072] h att = Cross_att(h conv ,h lstm ,h lstm )

[0073] h att represents the output of cross attention;

[0074] The attention output is then added to the encoder and fed into layer normalization:

[0075] h l_n =nn.layernorm(h att +dropout(h conv ))

[0076] After the normalization layer, a linear layer is first connected, then a ReLU activation function is passed, and then another linear layer; finally, the output of the linear layer is compared with h l_n Add together to get the final output representation of the encoder h out , calculated as follows:

[0077] h out=nn.laynorm(fc(max(0,fc(h l_n )))+dropout(h l_n ))

[0078] fc represents the linear layer, dropout (h l_n ) represents the output h of the normalization layer l_n Apply the Dropout operation to reduce the model's dependence on the output features of the normalization layer;

[0079] Step 2.4: Output h obtained by Step 2.3 out At the same time, it is sent to the decoder and speaker classifier module, and the decoder is used to predict the speech; in the speaker classifier module, h out First, it passes through the gradient inversion module to ensure that the output of the gradient inversion module remains unchanged during forward propagation, then passes through two linear layers, and finally uses the Softmax function to achieve speaker classification prediction.

[0080] Furthermore, the specific steps of step 3 are:

[0081] Step 3.1, deploy the trained ".pt" format model to the server, and implement the function of multi-user concurrent requests by calling the model, and build a Lao-English mixed language speech synthesis device, which includes a client and a server;

[0082] Step 3.2: The user inputs the original Lao text to be synthesized into the synthesis device client, and the client passes the input text to the translation model on the server; the model is responsible for synthesizing the input text into the target speech and returning the synthesized speech to the client interface for playback.

[0083] The device for speech synthesis of Old and English mixed languages ​​enhanced by phoneme knowledge comprises: a module for executing the method for speech synthesis of Old and English mixed languages ​​enhanced by phoneme knowledge.

[0084] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the Old-English mixed language speech synthesis method with enhanced phoneme knowledge is implemented.

[0085] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the Old-English mixed language speech synthesis method enhanced with phoneme knowledge.

[0086] A computer program product comprises a computer program, which, when executed by a processor, implements the Old-English mixed language speech synthesis method enhanced with phoneme knowledge.

[0087] In order to verify the effect of the Lao-English mixed language speech synthesis method and device enhanced with phoneme knowledge proposed in the present invention, the present invention conducted experiments on the English public dataset LJSpeech and the precisely marked Lao dataset, using an audio size of 22.05kHz as input, and the intermediate output was a Mel-spectrogram with a feature dimension size of 80. Since there are only 2,000 Lao corpus data, and the LJSpeech corpus has tens of thousands of corpora, in order to reduce the deviation of the model parameters in the English language and cause the model to perform poorly in Lao, the present invention uses 2,000 English corpora as input. The size ratio of the training set and the validation set for Lao and English is 4:1, and the parameters are set to β1=0.9, β2=0.999, ε=10 -6 , Adam optimizer, learning rate is 10 -3 , set the weight of guiding attention loss ratio α=1.

[0088] Experiment 1: Comparison of experimental results of different models in different languages;

[0089] In the experiment, the present invention uses a mixed language speech synthesis model based on Tacotron2 to compare with the improved model of the present invention (PP-Tacotron2).

[0090] Table 2 shows the objective performance of the baseline model and the improved model

[0091]

[0092] Among them, Lua represents the language (En for English and Lao for Lao), and Speaker is the designated speaker (en-s for English speakers and lao-s for Lao speakers);

[0093] The experiment is shown in Table 2. The improved model is better than the Tacotron2 model as a whole, with relatively lower MCD and RMSE values. The improvement in the synthesis effect on Lao speech is significantly greater than that on English. This is because after adding the fine-grained phoneme pronunciation knowledge, the model further strengthens the modeling of phoneme pronunciation knowledge, and can more easily see the phoneme-to-audio pronunciation knowledge, so that the audio quality of both single Lao language synthesis and Lao-English mixed language synthesis has been improved to a certain extent. It further shows that the introduced fine-grained phoneme pronunciation knowledge is conducive to the modeling of phoneme pronunciation, which also verifies that the model of the present invention is more effective in realizing text-to-pronunciation modeling.

[0094] Experiment 2: ablation experiment;

[0095] Figure 3 is the subjective MOS evaluation score of the ablation experiment of the present invention, Figure 3It shows that when only the phoneme fine-grained pronunciation embedding is used as the output of an encoder to be fused to the text convolutional encoder, whether it is splicing or attention mechanism fusion, the phoneme hidden state features output by the text encoder will be disturbed and the hidden state features will be damaged, resulting in the model decoder being unable to learn accurate pronunciation knowledge in the modeling of text to audio, and the model still cannot achieve satisfactory results even if it is trained for 300 cycles. Therefore, the present invention proposes a fused LSTM neural network module, and the fine-grained pronunciation features are sent to the LSTM neural network to extract deep features. The model based on the LSTM network, whether it adopts splicing or cross-attention fusion of the text encoder hidden state method, is better than directly using the original fine-grained pronunciation feature fusion effect, and only 50 cycles are needed to train to obtain a satisfactory model effect. At the same time, the method based on cross-attention has achieved the best model performance in Lao synthesis and Lao speaker mixed language synthesis of English.

[0096] Since the model without LSTM has poor effect, the present invention further conducts an objective evaluation of MCD and RMSE on the improved model with LSTM network.

[0097] Table 3 shows the ablation experiment results.

[0098]

[0099] The experimental results show that the model using the cross-attention fusion method generates audio with richer information in terms of objective evaluation indicators. Compared with the splicing method, the MCD value and RMSE value of the cross-attention fusion method are reduced by 1.74 and 2.187 respectively. Although the splicing method has enhanced the modeling of local time series, it performs poorly in global feature modeling, so the generated speech quality is not as good as the cross-attention fusion method.

[0100] Experiment 3: Exemplarization analysis;

[0101] In order to demonstrate the effect of the PP-Tacotron2 model on the speaker's mixed language pronunciation, the present invention sets up an example analysis experiment. Figure 4 The PP-Tacotron2 and the baseline model Tacotron2 are shown to predict the mel-spectrogram of English using Lao speakers. The English text sequence is used as input: "The weather is very sunny", which means "the weather is very sunny" in Chinese. Figure 4 (a) and Figure 4 (b) English audio synthesized by an English speaker and English audio synthesized by a Lao speaker using the PP-Tacotron2 model proposed in the present invention, respectively. Figure 4 (c) and Figure 4(d) English audio synthesized by English speakers and Lao speakers using the baseline model Tacotron2. Figure 4 (a) and Figure 4 (c) We can see that both models perform very well in the task of synthesizing English text from English speakers. This is because English speakers will not encounter phonemes they have never seen before and have learned pronunciation knowledge. Figure 4 (b) and Figure 4 (d) It can be seen that the use of PP-Tacotron2 and Tacotron2 models to achieve Laotian people speaking English has completely different effects. Figure 4 (a) Figure 4 (b) Figure 4 (c) The orange, red, and blue boxes in the three figures clearly show the modeling effect of PP-Tacotron2 in realizing the pronunciation of phonemes when Lao people speak English, especially in terms of pronunciation duration and phoneme pronunciation details. Figure 4 (d) shows that the baseline model Tacotron2 is rather sloppy in modeling the pronunciation of Lao people speaking English. The inability to accurately model the phoneme pronunciation knowledge leads to the loss of some details of the Mel-spectrogram, especially in the judgment of the stop label, which makes the model unable to predict the pronunciation duration and pronunciation feature information. Experiments have shown that the method based on the embedding of phoneme fine-grained pronunciation features provides speaker-independent pronunciation knowledge, which is conducive to the speaker's effective pronunciation of phonemes in other languages ​​in mixed language synthesis.

[0102] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for speech synthesis of an Old-English mixed language enhanced with phoneme knowledge, characterized in that: The method comprises: Step 1: Construct a phoneme fine-grained pronunciation knowledge dictionary; Step 2: Construct an English-Chinese mixed language speech synthesis model enhanced with phoneme knowledge. The English-Chinese mixed language speech synthesis model enhanced with phoneme knowledge includes: Fine-grained pronunciation features are incorporated in the encoding stage to directly model pronunciation; When training the model, the characters are first converted into IDs through the phoneme fine-grained pronunciation knowledge dictionary, and then a vector representation is generated through the embedding layer to obtain the traditional phoneme embedding. Then, the pronunciation features of the phonemes are represented as a one-dimensional matrix, and the differences in phoneme pronunciations are distinguished by setting the values ​​in the matrix to 1 or -1, thereby modeling the fine-grained pronunciation features. Step 3, Lao-English mixed language speech synthesis: The Lao-English mixed language speech synthesis model enhanced with phoneme knowledge in step 2 is integrated into an API interface that can be called on the server. By calling the model API interface, a Lao-English mixed language speech synthesis device is constructed for Lao-English mixed language speech synthesis.

2. A method for speech synthesis of an Old-English mixed language enhanced with phoneme knowledge, characterized in that: In the step 1, the IPA international phonetic alphabet is used to uniformly represent Lao and English as phonemes, and a phoneme fine-grained pronunciation knowledge dictionary is constructed on all phoneme sequences.

3. The Lao-English mixed language speech synthesis method based on phoneme fine-grained pronunciation knowledge enhancement according to claim 1 is characterized in that: The specific steps of step 1 include: Step 1.1 Phoneme dictionary construction: The Lao data crawled from the Internet were manually annotated, including the removal of useless characters; then, the Lao and English texts were converted into phoneme sequences using Epitran and Espeak tools, respectively, to preliminarily construct a phoneme dictionary; Step 1.2 Processing of phoneme fine-grained pronunciation knowledge: Based on the phoneme dictionary constructed in step 1.1, symbols and attribute tags are added so that each character has additional attributes, and the pronunciation of different characters is marked by different attribute values.

4. The Lao-English mixed language speech synthesis method based on phoneme fine-grained pronunciation knowledge enhancement according to claim 1 is characterized in that: In step 2, the operation process of the Old-English mixed language speech synthesis model enhanced with phoneme knowledge includes: Step 2.1: Based on the phoneme fine-grained pronunciation knowledge dictionary constructed in step 1, a one-hot encoding method is used to construct a one-dimensional all-zero matrix of equal length according to the size of the attribute dictionary. According to the attribute value of each phoneme marked in the fine-grained pronunciation dictionary, the corresponding label position in the one-dimensional all-zero matrix is ​​set to 1, and the opposite type is set to -1 to increase the gap between phonemes with far different pronunciations, thereby embedding the phoneme fine-grained pronunciation attributes, and then introducing the LSTM network to output the hidden state of the phoneme sequence on the fine-grained pronunciation embedding; the size of the input fine-grained pronunciation feature vector is: shape(Input vector )=B*L*H1; Among them, Input vector It is represented as the phoneme fine-grained pronunciation feature, B is the batch size, L is the longest sentence length in a batch, and H1 represents the embedding dimension of each character; The phoneme fine-grained pronunciation feature vector is passed through the LSTM network to obtain the fine-grained pronunciation hidden state h lstm , the Cain state is expressed as: h lstm =LSTM(Input vector ); Step 2.2: Use a text convolutional encoder to directly convert characters into IDs according to the pronunciation dictionary, and then obtain the phoneme embedding h through embedding. conv , the embedding dimension is 256; Step 2.3: Based on the hidden state values ​​of the two phoneme granularities obtained in Step 2.1 and Step 2.2, the fine-grained pronunciation hidden state h is converted into lstm Fused into the output vector h of the text encoder conv In order to increase the feature information of the encoder output; finally, the output of the encoder is further processed by the residual network structure, in which the fine-grained pronunciation hidden state h lstm As the cross-attention k and v, the output vector h of the text encoder conv As q, it is calculated as follows: h att =Cross_att(h conv ,h lstm ,h lstm ) h att represents the output of cross attention; The attention output is then added to the encoder and fed into layer normalization: h l_n =nn.layernorm(h att +dropout(h conv )) After the normalization layer, a linear layer is first connected, then a ReLU activation function is passed, and then another linear layer; finally, the output of the linear layer is compared with h l_n Add together to get the final output representation of the encoder h out , calculated as follows: h out =nn.laynorm(fc(max(0,fc(h l_n )))+dropout(h l_n )) fc represents the linear layer, dropout (h l_n ) represents the output h of the normalization layer l_n Apply the Dropout operation to reduce the model's dependence on the output features of the normalization layer; Step 2.4: Output h obtained by Step 2.3 out At the same time, it is sent to the decoder and speaker classifier module, and the decoder is used to predict the speech; in the speaker classifier module, h out First, it passes through the gradient inversion module to ensure that the output of the gradient inversion module remains unchanged during forward propagation, then passes through two linear layers, and finally uses the Softmax function to achieve speaker classification prediction.

5. The method for speech synthesis of Old and English mixed languages ​​enhanced with phoneme knowledge according to claim 1, characterized in that: The specific steps of step 3 are: Step 3.1, deploy the trained ".pt" format model to the server, and implement the function of multi-user concurrent request by calling the model, and build a Lao-English mixed language speech synthesis device, which includes a client and a server; Step 3.2: The user inputs the original Lao text to be synthesized into the synthesis device client, and the client passes the input text to the translation model on the server; the model is responsible for synthesizing the input text into the target speech and returning the synthesized speech to the client interface for playback.

6. An Old-English mixed language speech synthesis device enhanced with phoneme knowledge, characterized in that: include: A module for executing the Old-English mixed language speech synthesis method enhanced with phoneme knowledge as claimed in any one of claims 1 to 5.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the Old-English mixed language speech synthesis method with enhanced phoneme knowledge as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the Old-English mixed language speech synthesis method with enhanced phoneme knowledge is implemented as claimed in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the Old-English mixed language speech synthesis method with enhanced phoneme knowledge is implemented as claimed in any one of claims 1 to 5.

Images