Lao character transphoneme method based on transfer learning

Abstract

The invention relates to a method for converting Lao characters into phonemes based on transfer learning, and aims to realize efficient phoneme conversion by utilizing a model pre-trained on other languages and finely adjusting limited Lao data. According to the method, the problem of data scarcity can be relieved, the training process of the model can be accelerated, and the generalization ability of the model on new tasks is improved. A more robust Lao character transphoneme model can be constructed through transfer learning, and a foundation is laid for subsequent speech synthesis. As Thai with high language similarity with Lao is also a low-resource language and cannot be used as a pre-training language, Chinese is selected as the pre-training language. A Chinese open source data set is firstly used for training a Chinese G2P model based on a Transform architecture, decoding end parameters of the model are initialized to a student model, fine tuning training is performed by using a Lao data set with the scale of ten thousand, and the accuracy of a Lao character-to-phoneme model is improved.

Description

Technical Field

[0001] The invention relates to a Laotian character-to-phoneme method based on transfer learning, and belongs to the technical field of speech synthesis. Technical Background

[0002] Character-to-phoneme technology is an indispensable part of speech synthesis systems, which can convert text into natural spoken pronunciation. In the field of contemporary speech synthesis, the use of sequence-to-sequence models has become an advanced technical means. For character-to-phoneme tasks, sequence-to-sequence models are used to predict the pronunciation representation of the entire input text sequence, rather than being limited to the pronunciation of isolated words. This type of method significantly improves the accuracy of speech conversion of input text by capturing a wider range of contextual information, and effectively reduces the pronunciation inaccuracy caused by spelling errors. However, compared with traditional dictionary-based pronunciation prediction methods, sequence-to-sequence models rely on larger data sets for effective training and generalization, because complex neural network models with many parameters require more training data to ensure the reliability and stability of their prediction performance. Different languages ​​have their own unique phonemes and pronunciation rules, and researchers need to design conversion systems that can handle these unique phenomena.

[0003] Lao is a writing system derived from the Latin alphabet. The mapping relationship between phonemes and characters is complex and is influenced by many languages ​​and dialects. The unique phonemes in Lao are and Therefore, developing an efficient and accurate Lao character-to-phoneme system is of great significance for improving the application of language technology in the region. Summary of the invention

[0004] The problem to be solved by the present invention is: the present invention provides a Lao character to phoneme conversion method based on transfer learning, which aims to use models pre-trained on other languages ​​and achieve efficient phoneme conversion by fine-tuning limited Lao data. This method can not only alleviate the problem of data scarcity, but also accelerate the training process of the model and improve the generalization ability of the model on new tasks.

[0005] The present invention constructs a more robust Lao character-to-phoneme model through transfer learning, laying the foundation for subsequent speech synthesis. The Transformer architecture is a successful general sequence-to-sequence model designed for neural machine translation. Since G2P can be regarded as a translation problem, translating graphemes into their phonetic counterparts, the Transformer architecture can be applied to the G2P problem. Since Thai, which has a high linguistic similarity with Lao, is also a low-resource language and cannot be used as a pre-training language, the present invention selects Chinese as a pre-training language. Based on the Transformer architecture, a Chinese G2P model is first trained using a Chinese open source dataset, the decoding end parameters of the model are initialized to the student model, and a Lao dataset of 10,000 is used for fine-tuning training to improve the accuracy of the Lao character-to-phoneme model.

[0006] The technical solution of the present invention is: a Lao character phoneme conversion method for transfer learning, comprising:

[0007] Step 1: Data selection and preprocessing; select the CPP Chinese-Lao self-constructed dataset;

[0008] Step 2: Capture the semantic relationship and contextual information between characters through the Transformer encoder module;

[0009] Step 3: Fine-tune the Transformer model through the fine-tuning module: introduce a pre-trained Chinese Transformer model, and replace the embedding layer and output fully connected layer of the decoder part to adapt to different output phoneme set size requirements; finally, fine-tune the adjusted model under the same language and training dictionary scale conditions; use the fine-tuned Transformer model to convert Lao characters into phonemes.

[0010] As a further solution of the present invention, in Step 1, the number of words in the dictionary used to generate training data is limited; when testing the model, all words in the dictionary are used to generate test data, and sentences containing words not included in the dictionary are excluded; in addition, the frequency of occurrence of each word in the training set is counted, and based on this frequency information, it is decided which words to add to the small dictionary.

[0011] As a further solution of the present invention, the Step 2 includes:

[0012] The BERT pre-trained language model with Lao character granularity is used as part of the Transformer encoder module; the input character sequence is first processed by a tokenizer, which includes converting the text to lowercase, removing punctuation, and performing word segmentation; BERT uses the WordPiece word segmentation method to further subdivide words into subwords to optimize the size of the vocabulary and improve the generalization ability of the model; the introduction of the BERT pre-trained language model aims to make full use of character-level information to enhance the ability to cope with the challenge of lack of data in the Lao character-to-phoneme task; the feature extraction and representation learning capabilities of the BERT model are integrated into the character-to-phoneme model to better capture the semantic relationship and contextual information between characters.

[0013] As a further solution of the present invention, Step 3 includes:

[0014] The pre-trained Transformer model was fine-tuned. During the training of the Transformer model, the hyperparameters of all models were kept consistent to ensure the reliability of the experiment. The adopted Transformer model contained a total of 4.1M parameters during the training process. Through pre-training on the Chinese CPP dataset and 100 epochs of iterative training, the model state with the most ideal validation loss was saved. Subsequently, the Transformer model was trained from the initial state on a mixed dataset combining Chinese and Lao. The size range of the dictionary was adjusted, and after 50 epochs of training, the model with the smallest loss on the validation set was selected for further evaluation.

[0015] As a further solution of the present invention, Step 3 includes:

[0016] A pre-trained Chinese Transformer model was introduced, and the embedding layer and output fully connected layer of the decoder were replaced to adapt to different output phoneme set size requirements; finally, the adjusted model was fine-tuned under the same language and training dictionary size conditions, and the model with the best verification loss at the end of training was selected again for subsequent analysis and application.

[0017] The beneficial effects of the present invention are:

[0018] 1. This paper proposes a method for converting Lao characters into phonemes based on transfer learning, which aims to use models pre-trained on other languages ​​and achieve efficient phoneme conversion by fine-tuning limited Lao data. This method can not only alleviate the problem of data scarcity, but also accelerate the model training process and improve the model's generalization ability on new tasks;

[0019] 2. A more robust Lao character-to-phoneme model can be constructed through transfer learning, laying the foundation for subsequent speech synthesis. The Transformer architecture is a successful general sequence-to-sequence model designed for neural machine translation. Since G2P can be regarded as a translation problem, translating graphemes into their phonetic counterparts, the Transformer architecture can be applied to the G2P problem. Since Thai, which has a high linguistic similarity to Lao, is also a low-resource language and cannot be used as a pre-training language, the present invention selects Chinese as the pre-training language. Based on the Transformer architecture, a Chinese G2P model is first trained using a Chinese open source dataset, the decoding end parameters of the model are initialized to the student model, and a Lao dataset of 10,000 is used for fine-tuning training to improve the accuracy of the Lao character-to-phoneme model;

[0020] 3. The present invention achieves a significant improvement in model performance by pre-training the Transformer model on a Chinese dataset and then fine-tuning it using a Lao dataset. Compared with the model trained using only the Lao dataset, the fine-tuned model reduced the word error rate (WER) by 13.25% and the phoneme error rate (PER) by 3.15%. The final word error rate was 21.67% and the phoneme error rate was 4.51%. The experimental results verify the effectiveness of the pre-training combined with fine-tuning strategy in the character-to-phoneme task in low-resource scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a general design block diagram of a method for converting Lao characters into phonemes based on transfer learning proposed by the present invention;

[0022] Figure 2 A Laotian speech synthesis character-to-phoneme encoder diagram proposed by the present invention; DETAILED DESCRIPTION

[0023] The following is a further description of a Lao character phoneme conversion method based on transfer learning proposed by the present invention in conjunction with the accompanying drawings. It should be noted that the accompanying drawings are in a very simplified form, which is only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0024] Example 1: Figure 1 As shown, a method for converting Lao characters into phonemes based on transfer learning includes:

[0025] Step 1: Data selection and preprocessing: Select the CPP Chinese-Lao self-constructed dataset; CPP is an open source Chinese polyphone dataset. For the purpose of the experiment, the CPP Chinese-Lao self-constructed dataset was selected as the training and test data for the character-to-phoneme model. In order to make the language model more in line with actual application scenarios, the number of words in the dictionary used to generate training data was restricted. When testing the model, all the words in the dictionary were used to generate test data, and sentences containing words not included in the dictionary were excluded. In addition, to ensure that even a small subset selected from a large dictionary contains some high-frequency commonly used words, the frequency of occurrence of each word in the training set was counted. Based on this frequency information, decide which words to add to the small dictionary;

[0026] Step 2: Capture semantic relationships and contextual information between characters through the Transformer encoder module; Step 2 includes:

[0027] Since we focus on the character-to-phoneme task under scarce resources, in order to overcome the problem of data scarcity in the Lao character-to-phoneme task, we introduce more text features to assist the prediction of phoneme sequences. The BERT pre-trained language model with Lao character granularity is used as part of the Transformer encoder module. The input character sequence is first processed by the tokenizer, which includes converting the text to lowercase, removing punctuation, and performing word segmentation. BERT uses the WordPiece word segmentation method to further subdivide words into subwords to optimize the size of the vocabulary and improve the generalization ability of the model. Figure 2 As shown in the figure, by introducing the BERT pre-trained language model, it aims to make full use of character-level information to enhance the ability to cope with the challenge of lack of data in the Lao character-to-phoneme task; the feature extraction and representation learning capabilities of the BERT model are integrated into the character-to-phoneme model to better capture the semantic relationships and contextual information between characters.

[0028] Step 3: Fine-tune the Transformer model through the fine-tuning module: introduce the pre-trained Chinese Transformer model, replace the embedding layer and output fully connected layer of the decoder part to adapt to different output phoneme set size requirements; finally, fine-tune the adjusted model under the same language and training dictionary scale conditions; use the fine-tuned Transformer model to convert Lao characters into phonemes. Step 3 includes:

[0029] In order to make the pre-trained Transformer model better applicable to the Lao character-to-phoneme task, the pre-trained Transformer model was fine-tuned; during the training of the Transformer model, the hyperparameters of all models were kept consistent to ensure the reliability of the experiment; the adopted Transformer model contained a total of 4.1M parameters during the training process. Through pre-training on the Chinese CPP dataset, after 100 epochs of iterative training, the model state with the most ideal validation loss was saved; then, the Transformer model was trained from the initial state on a mixed dataset combining Chinese and Lao; in order to meet the needs of different dictionary sizes, the dictionary size range was adjusted, including [1K, 3K, 5K, 10K] words. After 50 epochs of training, the model with the smallest loss on the validation set was selected for further evaluation.

[0030] A pre-trained Chinese Transformer model was introduced, and the embedding layer and output fully connected layer of the decoder were replaced to adapt to different output phoneme set size requirements; finally, the adjusted model was fine-tuned for 25 epochs under the same language and training dictionary size conditions, and the model with the best verification loss at the end of training was selected again for subsequent analysis and application.

[0031] The present invention designs comparative experiments with other schemes to demonstrate the effectiveness of the proposed method. First, a Transformer-based model is used to independently train on a self-constructed Lao character-to-phoneme dataset. In order to evaluate the performance of the model in a data resource-constrained environment and its potential room for improvement, a series of comparative experiments are designed. In particular, a comparative analysis is conducted between a separately trained Transformer model and a model pre-processed by transfer learning to explore the gain effect of fine-tuning the pre-trained Transformer model on the character-to-phoneme task in a low-resource language environment.

[0032] In addition, in order to comprehensively evaluate the performance of the proposed method, it is also compared with traditional baseline models such as Recurrent Neural Network (RNN) and Bidirectional Long Short-Term Memory (BiLSTM). These models are selected as baseline models because they have been widely studied and used in sequence processing tasks. Through this comparison, the aim is to reveal the advantages and potential limitations of the Transformer structure compared to other mainstream models in processing character-to-phoneme tasks. The experimental results are shown in Table 1.

[0033] Table 1 Word error rate and phoneme error rate performance of different models. The Chinese dataset here is the entire CPP dataset.

[0034]

[0035]

[0036] From the experimental data, we can observe that among all models, when training data is scarce, whether it is RNN, BiLSTM or Transformer, their performance is not satisfactory. However, when the data set is sufficient, for the task of converting Chinese characters to phonemes, all models can achieve relatively ideal performance.

[0037] It is particularly noteworthy that the character-to-phoneme model obtained by pre-training the Transformer model on the Chinese dataset and then fine-tuning it using the Lao dataset showed significant performance improvements. Although its performance did not surpass the Chinese character-to-phoneme model, compared with the model trained only on the Lao dataset, the model achieved a significant decrease of 13.25% in word error rate (WER) and 3.15% in phoneme error rate (PER). This result reflects the effectiveness of the pre-training plus fine-tuning strategy in low-resource language processing.

[0038] In order to test whether the proposed transfer learning method can improve the performance of character-to-phoneme tasks in different languages, experiments were conducted on different languages ​​in the same way. Due to the high cost of acquiring low-resource languages, 10,000 data items of the same size as the Laotian data set were taken from the English, Japanese, Korean, and German data sets for experiments. In this way, they can also be regarded as low-resource languages ​​at the data set level. The experimental results are shown in Table 2:

[0039] Table 2 shows the performance of the proposed method on different language datasets. The size of each language dataset is 10,000 texts. w / indicates fine-tuning training by pre-training the Transformer on the Chinese dataset.

[0040] w / o means training directly on Transformer

[0041]

[0042]

[0043] When analyzing Table 2, it is found that there are significant differences in the performance obtained when using the transformer model for training with datasets of the same size, including English, Japanese, Korean, German, and Lao. Lao, in particular, lags behind in performance, which may be attributed to its unique writing system and the complexity of pronunciation characteristics, which makes it difficult for the model to capture its regularities, making modeling more difficult compared to other mainstream languages. From this, a preliminary conclusion can be drawn: the existing general character-to-phoneme method is not suitable for direct application to Lao.

[0044] Furthermore, when the proposed method is used, that is, when the model is trained using the pre-trained Chinese Transformer, it is found that the performance of all mainstream languages ​​has been improved. By analyzing the WER (word error rate) and PER (phoneme error rate) indicators in detail, it is observed that the performance of the Japanese and Korean character-to-phoneme models has improved more significantly than that of English and German. This result may be due to historical and cultural connections. Chinese, Japanese, and Korean have many commonalities in terms of language and writing. Since Japanese and Korean contain a large number of Chinese loanwords, the pre-trained model trained using the Chinese dataset can provide more useful information when performing Japanese and Korean transfer learning, thereby more effectively promoting model learning and optimization. Based on this hypothesis, the Transformer model was pre-trained with an English dataset and fine-tuned on a Lao dataset. The performance of the fine-tuned Chinese pre-trained model on the Lao character-to-phoneme task was compared. At the same time, a German dataset of the same size as the Lao dataset was taken to compare the performance of the pre-trained Chinese Transformer model and the pre-trained English Transformer model after fine-tuning on the German dataset. To ensure the objectivity of the experimental results, the size of the English dataset used was consistent with that of the Chinese dataset. The experimental results are shown in Table 3.

[0045] Table 3 Verification of the impact of language commonality on the character-to-phoneme model

[0046]

[0047] According to linguistic conclusions, English and German both belong to the West Germanic branch of the Germanic language family of the Indo-European language family and have a high degree of language similarity. Analyzing the experimental results shown in Table 2.4, it is found that when a small amount of German dataset is used to fine-tune the pre-trained English Transformer model, the performance of the German character-to-phoneme model is significantly improved, with the WER value reduced to 18.74%, and the PER value is only 3.95%. In contrast, fine-tuning the English pre-trained Transformer model with the Lao dataset and fine-tuning the Chinese pre-trained Transformer model with the German dataset, although it can improve the performance of the model to a certain extent, does not achieve the significant performance improvement that language similarity may bring. This result confirms the method proposed in this study: selecting the Chinese dataset as the pre-training dataset has a more significant effect on improving the performance of the Lao character-to-phoneme model.

[0048] The specific implementation modes of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A Lao character phoneme conversion method based on transfer learning, characterized in that: include: Step 1: Data selection and preprocessing; select the CPP Chinese-Lao self-constructed dataset; Step 2: Capture the semantic relationship and contextual information between characters through the Transformer encoder module; Step 3: Fine-tune the Transformer model through the fine-tuning module: introduce a pre-trained Chinese Transformer model, and replace the embedding layer and output fully connected layer of the decoder part to adapt to different output phoneme set size requirements; finally, fine-tune the adjusted model under the same language and training dictionary scale conditions; use the fine-tuned Transformer model to convert Lao characters into phonemes.

2. According to the method of Lao character phoneme conversion based on transfer learning in claim 1, it is characterized by: In the Step 1, the number of words in the dictionary used to generate training data is limited; when testing the model, all words in the dictionary are used to generate test data, and sentences containing words not included in the dictionary are excluded; in addition, the frequency of occurrence of each word in the training set is counted, and based on this frequency information, it is decided which words to add to the small dictionary.

3. According to the method of Lao character phoneme conversion based on transfer learning in claim 1, it is characterized by: The Step 2 includes: The BERT pre-trained language model with Lao character granularity is used as part of the Transformer encoder module; the input character sequence is first processed by a tokenizer, which includes converting the text to lowercase, removing punctuation, and performing word segmentation; BERT uses the WordPiece word segmentation method to further subdivide words into subwords to optimize the size of the vocabulary and improve the generalization ability of the model; the introduction of the BERT pre-trained language model aims to make full use of character-level information to enhance the ability to cope with the challenge of lack of data in the Lao character-to-phoneme task; the feature extraction and representation learning capabilities of the BERT model are integrated into the character-to-phoneme model to better capture the semantic relationship and contextual information between characters.

4. According to the method of Lao character phoneme conversion based on transfer learning as claimed in claim 1, it is characterized by: The Step 3 includes: The pre-trained Transformer model was fine-tuned. During the training of the Transformer model, the hyperparameters of all models were kept consistent to ensure the reliability of the experiment. The adopted Transformer model contained a total of 4.1M parameters during the training process. Through pre-training on the Chinese CPP dataset and 100 epochs of iterative training, the model state with the most ideal validation loss was saved. Subsequently, the Transformer model was trained from the initial state on a mixed dataset combining Chinese and Lao. The size range of the dictionary was adjusted, and after 50 epochs of training, the model with the smallest loss on the validation set was selected for further evaluation.

5. According to the method of Lao character phoneme conversion based on transfer learning as claimed in claim 1, it is characterized by: The Step 3 includes: A pre-trained Chinese Transformer model was introduced, and the embedding layer and output fully connected layer of the decoder were replaced to adapt to different output phoneme set size requirements; finally, the adjusted model was fine-tuned under the same language and training dictionary size conditions, and the model with the best verification loss at the end of training was selected again for subsequent analysis and application.

Images