Phoneme knowledge enhanced old-english mixed language speech synthesis method and apparatus

Abstract

The present application relates to a method and device for Old-English mixed language speech synthesis based on phoneme knowledge enhancement, and belongs to the technical field of natural language processing.The present application comprises the following steps: constructing a phoneme fine-grained pronunciation knowledge dictionary; constructing an Old-English mixed language speech synthesis model based on phoneme knowledge enhancement, wherein the Old-English mixed language speech synthesis model based on phoneme knowledge enhancement comprises: fine-grained pronunciation features are integrated in the coding stage to directly model the pronunciation; during model training, first, characters are converted into id identifiers through the phoneme fine-grained pronunciation knowledge dictionary, and then vector representation is generated through an embedding layer to obtain traditional phoneme embedding; then, the pronunciation features of phonemes are represented as a one-dimensional matrix, and the values in the matrix are set as 1 or -1 to distinguish the differences in phoneme pronunciation, thereby modeling the fine-grained pronunciation features; and Old-English mixed language speech synthesis.The present application can more accurately perform pronunciation synthesis.

Description

Technical Field

[0001] The present invention relates to a phoneme knowledge-enhanced Old-English mixed language speech synthesis method and device, belonging to the technical field of natural language processing. Background Art

[0002] With the development of artificial intelligence (AI) technology, the performance of acoustic and vocoder models for speech synthesis has continued to improve. Given sufficient data, single-language speech synthesis tasks can now produce relatively high-quality speech. However, as technology evolves, demand for mixed-language speech synthesis is gradually gaining attention. Mixed-language speech synthesis has matured in recent years, achieving excellent results in resource-rich languages. However, due to the occasional presence of English words in Laotian text, when speech synthesis is required for such text, Lao speakers often struggle to accurately pronounce English words. This is because the different representations of the mixed language make it difficult for Lao speakers to pronounce English phonemes.

[0003] Each language has its own unique phoneme set, which makes it difficult to achieve effective pronunciation of mixed-language phonemes. In practice, the unique phoneme sets of Lao and English make it very difficult to transfer speaker phonemes when synthesizing audio, reducing the fluency and naturalness of the entire model. For example, the phoneme " h " is a phoneme that does not exist in English. Therefore, when synthesizing Lao audio using English speakers, it is difficult to recognize the unseen phoneme" h "Achieve pronunciation." Therefore, in mixed-language speech synthesis tasks, achieving a unified representation of mixed languages ​​from text to pronunciation is crucial. Existing methods typically take text sequences or phoneme sequences as input. During the text or phoneme-to-sound conversion process, the model explicitly ignores the hidden phoneme features and simply synthesizes the phonemes into a fixed sound. If the hidden phoneme information can be explicitly included as one of the inputs in the input sequence, the model can perform deeper modeling of the explicit high-level features when implementing mixed-language speaker synthesis, especially for invisible phonemes in speaker transfer, thereby improving the naturalness and fluency of the synthesized audio during speaker transfer. Summary of the Invention

[0004] The present invention provides a Lao-English mixed language speech synthesis method and device enhanced with phoneme knowledge. By adding speaker-independent phoneme fine-grained pronunciation features as explicit pronunciation features and fusing the pronunciation features with encoder features, it helps Lao speakers to effectively pronounce English phonemes during model inference.

[0005] The technical solution of the present invention is: a phoneme knowledge-enhanced speech synthesis method for an Old-English mixed language, the method comprising:

[0006] Step 1, constructing a phoneme fine-grained pronunciation knowledge dictionary;

[0007] Step 2, constructing a phoneme knowledge enhanced Lao-English mixed language speech synthesis model, the phoneme knowledge enhanced Lao-English mixed language speech synthesis model comprising:

[0008] Fine-grained pronunciation features are integrated in the encoding stage to directly model the pronunciation;

[0009] In the model training, first, the characters are converted into id identifiers through the phoneme fine-grained pronunciation knowledge dictionary, and then the vector representation is generated through the embedding layer, so as to obtain the traditional phoneme embedding; then, the pronunciation features of the phonemes are represented as a one-dimensional matrix, and the values in the matrix are set to 1 or -1 to distinguish the differences in the pronunciation of the phonemes, so as to model the fine-grained pronunciation features; combining the phoneme pronunciation feature vector and the traditional phoneme embedding sequence, the model can accurately model the pronunciation of the phonemes of Lao and English in the training stage, which is used to alleviate the difficulty of unified modeling of mixed language phonemes;

[0010] Step 3, Lao-English mixed language speech synthesis: integrating the phoneme knowledge enhanced Lao-English mixed language speech synthesis model of step 2 into an API interface that can be called on a server, through calling the model API interface, a Lao-English mixed language speech synthesis device is constructed, which is used for Lao-English mixed language speech synthesis.

[0011] Further, in step 1, the IPA international phonetic alphabet is used to uniformly represent Lao and English as phonemes, and a phoneme fine-grained pronunciation knowledge dictionary is constructed on all phoneme sequences.

[0012] Further, the specific steps of step 1 include:

[0013] Step 1.1 phoneme dictionary construction: manually annotating the Lao data crawled from the Internet, including removing useless characters; then, using Epitran and Espeak tools to convert Lao and English text into phoneme sequences to preliminarily construct a phoneme dictionary;

[0014] Step 1.2 phoneme fine-grained pronunciation knowledge processing: based on the phoneme dictionary constructed in step 1.1 plus symbols and attribute labels, so that each character has additional attributes, and the pronunciation of different characters is marked by assigning different attribute values.

[0015] Further, in step 2, the operation process of the phoneme knowledge enhanced Lao-English mixed language speech synthesis model includes:

[0016] Step 2.1, based on the phoneme fine-grained pronunciation knowledge dictionary constructed in step 1, one-hot encoding is used, and a one-dimensional all-zero matrix of equal length is constructed according to the size of the attribute dictionary, and according to the attribute value marked in the fine-grained pronunciation dictionary of each phoneme, the corresponding label position in the one-dimensional all-zero matrix is 1, and the opposite type is-1, so as to intensify the difference between the phonemes with far pronunciation, so as to embed the phoneme fine-grained pronunciation attribute, and then introduce the LSTM network to output the hidden state of the phoneme sequence in the fine-grained pronunciation embedding; the size of the input fine-grained pronunciation feature vector is:

[0017] shape(Input vector )=B*L*H1;

[0018] Wherein, Input vector represents the phoneme fine-grained pronunciation feature, B is the batch size, L is the length of the longest sentence in a batch, and H1 represents the embedding dimension of each character;

[0019] The phoneme fine-grained pronunciation feature vector obtains the fine-grained pronunciation hidden state h lstm through the LSTM network, and the hidden state is represented as:

[0020] h lstm =LSTM(Input vector );

[0021] Step 2.2, the text convolutional encoder is used to directly convert the character to id according to the pronunciation dictionary, and then the phoneme embedding h conv is obtained through embedding, and the embedding dimension is 256;

[0022] Step 2.3, based on the hidden state values of the two phoneme granularities obtained in steps 2.1 and 2.2, the fine-grained pronunciation hidden state h lstm is fused into the output vector h conv of the text encoder through the cross-attention mechanism, so as to increase the feature information of the encoder output; finally, the residual network structure is used to further process the output of the encoder, wherein the fine-grained pronunciation hidden state h lstm is used as k and v of cross-attention, and the output vector h conv of the text encoder is used as q, and the calculation is as follows:

[0023] h att =Cross_att(h conv ,h lstm ,h lstm )

[0024] h att represents the output of cross-attention;

[0025] The attention output is added to the encoder and sent to the layer normalization:

[0026] h l_n = nn. layernorm(h att + dropout(h conv ))

[0027] After the normalization layer, a linear layer is first connected, and then a ReLU activation function is passed, followed by another linear layer; finally, the output of the linear layer is added to h l_n , thereby obtaining the final output representation h out of the encoder, which is calculated as follows:

[0028] h out = nn. laynorm(fc(max(0, fc(h l_n )) + dropout(h l_n ))

[0029] fc represents a linear layer, and dropout(h l_n ) represents a Dropout operation applied to the output h l_n of the normalization layer to reduce the model's dependence on the output features of the normalization layer.

[0030] Step 2.4, the output h out obtained in Step 2.3 is sent to the decoder and the speaker classifier module at the same time, and speech prediction is performed through the decoder; in the speaker classifier module, h out is first passed through a gradient reversal module to ensure that the output of the gradient reversal module remains unchanged during forward propagation, then passed through two linear layers, and finally classified and predicted by a Softmax function.

[0031] Further, the specific steps of step 3 are as follows:

[0032] Step 3.1, deploy the trained ".pt" format model to the server side, and realize the function of multiple user concurrent requests by calling the model, and build a Lao-English mixed language speech synthesis device, which includes a client and a server;

[0033] Step 3.2, the user inputs the original Lao text to be synthesized into the synthesis device client, and the client passes the input text to the translation model on the server side; the model is responsible for synthesizing the input text into target speech, and returns the synthesized speech to the client interface for playback.

[0034] The phoneme knowledge enhanced Lao-English mixed language speech synthesis device includes a module for executing the phoneme knowledge enhanced Lao-English mixed language speech synthesis method.

[0035] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method for old-English mixed language speech synthesis enhanced by phoneme knowledge when executing the program.

[0036] A non-transitory computer-readable storage medium having stored thereon a computer program, the computer program implementing the method for old-English mixed language speech synthesis enhanced by phoneme knowledge when executed by a processor.

[0037] A computer program product comprising a computer program, the computer program implementing the method for old-English mixed language speech synthesis enhanced by phoneme knowledge when executed by a processor.

[0038] The beneficial effects of the present application are:

[0039] The present application introduces invisible pronunciation feature information in phoneme embedding, each phoneme and symbol is marked with type and multiple pronunciation attributes, the model can see more information of the phoneme in the training stage, so that the pronunciation synthesis can be more accurate when converting the phoneme to audio, effectively alleviating the difficulty of unified modeling of mixed language phonemes and speaker timbre. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The flowchart in the present application;

[0041] Figure 2 The network architecture diagram of the old-English mixed language speech synthesis enhanced by phoneme knowledge in the present application;

[0042] Figure 3 The schematic diagram of subjective MOS evaluation score in the ablation experiment in the present application;

[0043] Figure 4 The Mel spectrogram change schematic diagram of the PP-Tacotron2 and the benchmark model in the present application when a Laotian speaks English. DETAILED DESCRIPTION

[0044] Embodiment 1: As shown in the old-English mixed language speech synthesis enhanced by phoneme knowledge and the device, the method comprises: Figures 1-4

[0045] Step 1, constructing a phoneme fine-grained pronunciation knowledge dictionary;

[0046] Step 2, constructing an old-English mixed language speech synthesis model enhanced by phoneme knowledge, the old-English mixed language speech synthesis model enhanced by phoneme knowledge comprises:

[0047] Fine-grained pronunciation features are integrated in the encoding stage, and the pronunciation is directly modeled; ​

[0048] During model training, the characters are first converted into IDs using a fine-grained phoneme pronunciation knowledge dictionary. A vector representation is then generated through an embedding layer, resulting in a traditional phoneme embedding. The phoneme pronunciation features are then represented as a one-dimensional matrix. Differences in phoneme pronunciation are distinguished by setting the matrix values ​​to 1 or -1, thereby modeling the fine-grained pronunciation features. Combining the phoneme pronunciation feature vectors with traditional phoneme embedding sequences allows the model to accurately model phoneme pronunciation in Lao and English during training, alleviating the difficulty of unified modeling of mixed-language phonemes.

[0049] Step 3, Lao-English mixed language speech synthesis: The Lao-English mixed language speech synthesis model enhanced with phoneme knowledge in Step 2 is integrated into an API interface that can be called on the server. By calling the model API interface, a Lao-English mixed language speech synthesis device is constructed for Lao-English mixed language speech synthesis.

[0050] Furthermore, in step 1, the IPA International Phonetic Alphabet is used to uniformly represent Lao and English as phonemes, and a phoneme fine-grained pronunciation knowledge dictionary is constructed based on all phoneme sequences.

[0051] Furthermore, the specific steps of step 1 include:

[0052] Step 1.1 Phoneme Dictionary Construction: Laotian data scraped from the internet were manually annotated, including removing useless characters. Subsequently, Epitran and Espeak were used to convert Laotian and English texts into phoneme sequences, respectively, to initially construct a phoneme dictionary.

[0053] Step 1.2 Processing phoneme pronunciation knowledge: Based on the phoneme dictionary constructed in step 1.1, symbols and attribute tags are added to give each character additional attributes. Different attribute values ​​are assigned to mark the pronunciation of different characters. For specific attributes, see Table 1. symbol_type indicates the type of each character. If it is a phoneme, it is marked as ph. If it is a symbol, it is marked with other attribute values, such as '?': {'symbol_type': 'qm'}, '!': {'symbol_type': 'em'}. Each phoneme has two fixed attributes: vowel_consonant and VUV. The vowel_consonant attribute divides phonemes into three categories: vowels, consonants, and others. VUV marks the pronunciation type as voiceless or voiced. Different phoneme categories affect other phoneme attributes.

[0054] For example, the consonant '∫' symbol in the phoneme sequence converted from the English dataset:

[0055]

[0056]

[0057] The individual phoneme symbols symbol_type are marked as ph type, and additional phoneme attributes are added, such as whether it is a vowel consonant vowel_consonant, a voiceless or voiced consonant VUV, and for consonants, there are two attributes: consonant position consonant_place and consonant manner consonant_manner.

[0058] For vowels, symbol:

[0059]

[0060] In addition to the symbol_type, vowel_consonant, and VUV attributes, each vowel also has vowel-specific attributes vowel_frontness, vowel_openness, and vowel_roundedness, which are fine-grained representations of the vowel's pronunciation, position, and roundness, respectively.

[0061] The present invention uses the above method to perform more fine-grained feature representation on all phonemes and symbols in English and Lao, thereby constructing an explicit phoneme fine-grained pronunciation knowledge dictionary.

[0062] Table 1 Phoneme fine-grained pronunciation attributes and attribute values

[0063]

[0064] Furthermore, in step 2, the operation process of the Old-English mixed language speech synthesis model enhanced with phoneme knowledge includes:

[0065] Step 2.1: Based on the phoneme fine-grained pronunciation knowledge dictionary constructed in Step 1, use one-hot encoding to construct a one-dimensional all-zero matrix of equal length according to the size of the attribute dictionary. According to the attribute value of each phoneme in the fine-grained pronunciation dictionary, the corresponding label position in the one-dimensional all-zero matrix is ​​set to 1, and the opposite type is set to -1 to increase the gap between phonemes with very different pronunciations, thereby embedding the phoneme fine-grained pronunciation attributes. Then, introduce an LSTM network to output the hidden state of the phoneme sequence on the fine-grained pronunciation embedding; the size of the input fine-grained pronunciation feature vector is:

[0066] shape(Input vector )=B*L*H1;

[0067] Among them, Input vectorrepresents the phoneme fine-grained pronunciation feature, B is the batch size, L is the length of the longest sentence in a batch, H1 represents the embedding dimension of each character, and H2 represents the embedding dimension of each phoneme;

[0068] The phoneme fine-grained pronunciation feature vector obtains a fine-grained pronunciation hidden state h lstm , which is represented as:

[0069] h lstm =LSTM(Input vector );

[0070] Step 2.2, directly convert characters into ids according to the pronunciation dictionary using a text convolutional encoder, and then obtain phoneme embeddings h conv through embedding, and the embedding dimension is 256;

[0071] Step 2.3, based on the two phoneme-grained hidden state values obtained in steps 2.1 and 2.2, the fine-grained pronunciation hidden state h lstm is fused into the output vector h conv of the text encoder through a cross-attention mechanism, thereby increasing the feature information of the encoder output; finally, a residual network structure is used to further process the output of the encoder, wherein the fine-grained pronunciation hidden state h lstm is used as k and v, and the output vector h conv of the text encoder is used as q, and the calculation is as follows:

[0072] h att =Cross_att(h conv ,h lstm ,h lstm )

[0073] h att represents the output of the cross-attention;

[0074] The attention output is added to the encoder and sent to the layer normalization:

[0075] h l_n =nn.layernorm(h att +dropout(h conv ))

[0076] After the normalization layer, a linear layer is first connected, then a ReLU activation function is passed, and then a linear layer is passed again; finally, the output of the linear layer is added to h l_n , thereby obtaining the final output representation h out of the encoder, and the calculation is as follows:

[0077] h out= nn. laynorm(fc(max(0, fc(h l_n )))+dropout(h l_n ))

[0078] fc denotes a linear layer, dropout(h l_n ) denotes a normalized layer output h l_n The Dropout operation is applied to reduce the dependence of the model on the normalized layer output features;

[0079] Step 2.4, the output h out is sent into the decoder and the speaker classifier module, the speech is predicted by the decoder; in the speaker classifier module, h out is first passed through the gradient reversal module to ensure that the gradient reversal module output remains unchanged during forward propagation, then passes through two linear layers, and finally realizes speaker classification prediction through the Softmax function.

[0080] Further, the specific steps of step 3 are:

[0081] Step 3.1, deploy the trained “.pt” format model to the server side, realize the function of multi-user concurrent request by calling the model, and build a Lao-English mixed language speech synthesis device, which includes a client and a server;

[0082] Step 3.2, the user inputs the original Lao text to be synthesized into the synthesis device client, and the client passes the input text to the translation model on the server side; the model is responsible for synthesizing the input text into target speech, and returns the synthesized speech to the client interface for playing.

[0083] The phoneme knowledge enhanced Lao-English mixed language speech synthesis device comprises a module for executing the phoneme knowledge enhanced Lao-English mixed language speech synthesis method.

[0084] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the phoneme knowledge enhanced Lao-English mixed language speech synthesis method.

[0085] A non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the phoneme knowledge enhanced Lao-English mixed language speech synthesis method.

[0086] A computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the phoneme knowledge enhanced Lao-English mixed language speech synthesis method.

[0087] In order to verify the effect of the phoneme knowledge enhanced Lao-English mixed language speech synthesis method and device proposed in the present application, experiments are carried out on the English public data set LJSpeech and the fine-labeled Lao language data set. The audio size of 22.05 kHz is used as input, and the intermediate output is a mel spectrogram with a feature dimension size of 80. Since the Lao language corpus data only has 2000, and the LJSpeech corpus set has tens of thousands of corpus, in order to reduce the influence of the deviation of the model parameters on the English language, which leads to poor performance of the model on the Lao language, the present application uses 2000 English corpus as input. The size ratio of the training set and the verification set of the Lao language and the English language is 4:1, the parameter setting is β1=0.9, β2=0.999, and ε=10 -6 , Adam optimizer, learning rate is 10 -3 , the proportion weight of the guide attention loss is set to α=1.

[0088] Experiment 1: Comparison of experimental results of different models under different languages

[0089] In the experiment, the present application uses a mixed language speech synthesis model based on Tacotron2 and the improved model (PP-Tacotron2) of the present application for comparison.

[0090] Table 2 is the objective performance of the reference model and the improved model

[0091]

[0092] Among them, Lua represents language (En English and Lao Lao language), Speaker is a specified speaker (en-s English speaker and lao-s Lao speaker);

[0093] As shown in Table 2, the improved model is better than the Tacotron2 model as a whole, and the values of MCD and RMSE are relatively lower, and the synthesis effect on the Lao language is obviously improved more than that on the English language. This is because after increasing the phoneme fine-grained pronunciation knowledge, the modeling of the model on the phoneme pronunciation knowledge is further strengthened, and the phoneme to audio pronunciation knowledge can be easily seen, so that the audio quality of single Lao language synthesis or Lao-English mixed language synthesis is improved. Further indicates that the introduction of phoneme fine-grained pronunciation knowledge is beneficial to the modeling of the model on the phoneme pronunciation, which also verifies that the model of the present application has better effect in realizing text to pronunciation modeling.

[0094] Experiment 2: Ablation experiment

[0095] Figure 3 The subjective MOS evaluation score of the ablation experiment of the present application is Figure 3It is shown that when only embedding the phoneme fine-grained pronunciation as an output of an encoder into the text convolutional encoder, whether using splicing or attention mechanism fusion, the phoneme hidden state features output by the text encoder will be disturbed and damaged, resulting in the model decoder being unable to learn accurate pronunciation knowledge in the modeling of text to audio, and the model still cannot achieve satisfactory results even after 300 cycles of training. Therefore, the present application proposes to fuse an LSTM neural network module, and send the fine-grained pronunciation features into the LSTM neural network to extract deep features. The model based on the LSTM network is better than directly using the original fine-grained pronunciation features in terms of fusion effect, whether using splicing or cross-attention fusion of text encoder hidden state, and only 50 cycles are needed to train a satisfactory model effect. At the same time, the cross-attention-based method achieves the best performance in Lao language synthesis and Lao speaker mixed language synthesis of English.

[0096] Since the model without adding LSTM has poor effect, the present application further objectively evaluates the improved model with LSTM network in terms of MCD and RMSE.

[0097] Table 3 is the ablation experiment result

[0098]

[0099] The experimental results show that the model using the cross-attention fusion method generates audio containing more rich information in terms of objective evaluation indicators. Compared with the splicing method, the MCD value and the RMSE value of the cross-attention fusion method are reduced by 1.74 and 2.187 respectively. Although the splicing method has some enhancement in local time sequence modeling, it performs poorly in global feature modeling, so the quality of the generated speech is not as good as that of the cross-attention fusion method.

[0100] Experiment three: example analysis

[0101] In order to reflect the effect of the PP-Tacotron2 model in speaker mixed language pronunciation, the present application sets up an example analysis experiment, Figure 4 The PP-Tacotron2 and the benchmark model Tacotron2 use Lao speakers to predict English mel-spectrograms. The English text sequence is used as input: "The weather is very sunny", which means "The weather is very sunny" in Chinese. Figure 4 (a) and Figure 4 (b) are the English audio synthesized by the PP-Tacotron2 model proposed in the present application using an English speaker and the English audio synthesized by the PP-Tacotron2 model using a Lao speaker, Figure 4 (c) and Figure 4(d) English speaker synthesized English audio and Lao speaker synthesized English audio are used for the baseline model Tacotron2 respectively. Comparison Figure 4 (a) and Figure 4 (c), it can be seen that the two models perform very well in the English speaker to English text synthesis task, because the English speaker will not encounter an unfamiliar phoneme, while learning pronunciation knowledge. Comparison Figure 4 (b) and Figure 4 (d), it can be seen that there are completely different effects of using PP-Tacotron2 and Tacotron2 models to realize Lao people speaking English, comparison Figure 4 (a), Figure 4 (b), Figure 4 (c) three figures, the orange box, red box, blue box, it can be clearly seen that the PP-Tacotron2 has a great effect on the modeling of phoneme pronunciation when Lao people speak English, especially in the details of phoneme pronunciation length and phoneme pronunciation. While Figure 4 Figure 4 (d) shows that the baseline model Tacotron2 is quite rough in modeling the pronunciation of Lao people speaking English, resulting in a part of the details missing in the mel-spectrogram, especially in the judgment of the stop label, which is particularly inaccurate, so that the model cannot predict better pronunciation length and pronunciation feature information. Experiments show that the method based on phoneme fine-grained pronunciation feature embedding provides speaker-independent pronunciation knowledge, which is beneficial to the effective pronunciation of other languages by the speaker in mixed language synthesis.

[0102] The specific embodiments of the application are described in detail above in conjunction with the accompanying drawings, but the application is not limited to the above-mentioned embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the application.

Claims

1. A method for speech synthesis of an Old-English mixed language enhanced with phoneme knowledge, characterized by: The method comprises: Step 1: Construct a phoneme fine-grained pronunciation knowledge dictionary; Step 2: Construct a phoneme-knowledge-enhanced mixed language speech synthesis model. The phoneme-knowledge-enhanced mixed language speech synthesis model includes: Fine-grained pronunciation features are incorporated into the encoding stage to directly model pronunciation; During model training, the characters are first converted into IDs using the phoneme fine-grained pronunciation knowledge dictionary. A vector representation is then generated through the embedding layer, resulting in a traditional phoneme embedding. The phoneme pronunciation features are then represented as a one-dimensional matrix. Differences in phoneme pronunciation are distinguished by setting the matrix values ​​to 1 or -1, thereby modeling the fine-grained pronunciation features. Step 3: Lao-English mixed language speech synthesis: The Lao-English mixed language speech synthesis model enhanced with phoneme knowledge in Step 2 is integrated into an API interface that can be called on the server. By calling the model API interface, a Lao-English mixed language speech synthesis device is built for Lao-English mixed language speech synthesis; In step 2, the operation process of the English-Chinese mixed language speech synthesis model enhanced with phoneme knowledge includes: Step 2.1: Based on the phoneme fine-grained pronunciation knowledge dictionary constructed in Step 1, use one-hot encoding to construct a one-dimensional all-zero matrix of equal length according to the size of the attribute dictionary. According to the attribute value of each phoneme in the fine-grained pronunciation dictionary, the corresponding label position in the one-dimensional all-zero matrix is ​​set to 1, and the opposite type is set to -1 to increase the gap between phonemes with very different pronunciations, thereby embedding the phoneme fine-grained pronunciation attributes. Then, introduce an LSTM network to output the hidden state of the phoneme sequence on the fine-grained pronunciation embedding; the size of the input fine-grained pronunciation feature vector is: ； in, It is represented as the phoneme fine-grained pronunciation feature, B is the batch size, L is the longest sentence length in a batch, represents the embedding dimension of each character; The phoneme fine-grained pronunciation feature vector is used to obtain the fine-grained pronunciation hidden state through the LSTM network. , the Cain state is expressed as: ； Step 2.2: Use a text convolutional encoder to directly convert characters into IDs according to the pronunciation dictionary, and then obtain phoneme embeddings through embedding , the embedding dimension is 256; Step 2.3: Based on the hidden state values ​​of the two phoneme granularities obtained in steps 2.1 and 2.2, the fine-grained pronunciation hidden state is added to the phoneme granularity through the cross attention mechanism. Fused into the output vector of the text encoder In order to increase the feature information of the encoder output; finally, the residual network structure is used to further process the output of the encoder, in which the fine-grained pronunciation hidden state As k and v of the cross attention, the output vector of the text encoder As q, it is calculated as follows: ； represents the output of cross attention; The attention output is then added to the encoder and fed into the layer normalization: ； After the normalization layer, a linear layer is first connected, then through the ReLU activation function, and then through another linear layer; finally, the output of the linear layer is compared with Add together to get the final output representation of the encoder , calculated as follows: ； represents the linear layer, Represents the output of the normalization layer Apply the Dropout operation to reduce the model's dependence on the output features of the normalization layer; Step 2.4: Output from step 2.3 At the same time, it is sent to the decoder and speaker classifier module, and the decoder is used to predict the speech; in the speaker classifier module, First, it passes through the gradient inversion module to ensure that the output of the gradient inversion module remains unchanged during forward propagation. Then it passes through two linear layers and finally uses the Softmax function to achieve speaker classification prediction.

2. The method for speech synthesis of an Old-English mixed language enhanced with phoneme knowledge according to claim 1, characterized in that: In the step 1, the IPA International Phonetic Alphabet is used to uniformly represent Lao and English as phonemes, and a phoneme fine-grained pronunciation knowledge dictionary is constructed based on all phoneme sequences.

3. The method for speech synthesis of an Old-English mixed language enhanced with phoneme knowledge according to claim 1, characterized in that: The specific steps of step 1 include: Step 1.1 Phoneme Dictionary Construction: Laotian data scraped from the internet were manually annotated, including removing useless characters. Subsequently, Epitran and Espeak were used to convert Laotian and English texts into phoneme sequences, respectively, to initially construct a phoneme dictionary. Step 1.2 Processing of phoneme fine-grained pronunciation knowledge: Based on the phoneme dictionary constructed in step 1.1, symbols and attribute tags are added so that each character has additional attributes, and the pronunciation of different characters is marked by different attribute values.

4. The method for speech synthesis of an Old-English mixed language enhanced with phoneme knowledge according to claim 1, characterized in that: The specific steps of step 3 are: Step 3.1: Deploy the trained ".pt" format model to the server. By calling the model, the function of multi-user concurrent requests is realized. A Lao-English mixed language speech synthesis device is built, which includes a client and a server. Step 3.2: The user inputs the original Lao text to be synthesized into the synthesis device client, and the client passes the input text to the translation model on the server; the model is responsible for synthesizing the input text into the target speech and returning the synthesized speech to the client interface for playback.

5. The speech synthesis device for mixed English-Chinese languages ​​enhanced with phoneme knowledge is characterized by: include: A module for executing the Old-English mixed language speech synthesis method enhanced with phoneme knowledge as claimed in any one of claims 1 to 4.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the Old-English mixed language speech synthesis method with enhanced phoneme knowledge as claimed in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the Old-English mixed language speech synthesis method with enhanced phoneme knowledge as claimed in any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the Old-English mixed language speech synthesis method with enhanced phoneme knowledge as claimed in any one of claims 1 to 4 is implemented.

Images