A laos language character-to-phoneme method based on transfer learning

Abstract

The present application relates to a kind of Laos character transphoneme method based on transfer learning, to utilize the model pre-trained on other language, and by fine-tuning to limited Laos data, to realize efficient phoneme conversion.This method can not only alleviate the problem of data scarcity, but also accelerate the training process of model, improve the generalization ability of model on new task.Through transfer learning, a more robust Laos character transphoneme model can be built, laying the foundation for subsequent speech synthesis.Because Thai language, which has high language similarity with Laos, is also a low-resource language, it cannot be used as a pre-training language, so Chinese is chosen as the pre-training language.Based on the Transformer architecture, a Chinese G2P model is first trained using the Chinese open-source dataset, the decoding end parameters of the model are initialized to the student model, and the model is fine-tuned and trained using a 10,000-size Laos dataset to improve the accuracy of the Laos character transphoneme model.

Description

TECHNICAL FIELD

[0001] The present application relates to a Laotian character-to-phoneme method based on transfer learning, belonging to the technical field of speech synthesis. TECHNICAL BACKGROUND

[0002] Character-to-phoneme technology is an indispensable part of speech synthesis systems, which can convert text into natural spoken pronunciation. In the field of contemporary speech synthesis, the use of sequence-to-sequence models has become an advanced technical means. For the character-to-phoneme task, the sequence-to-sequence model is used to predict the pronunciation representation of the entire input text sequence, rather than being limited to the pronunciation of isolated words. This method significantly improves the accuracy of speech conversion of input text by capturing more extensive context information, effectively reducing the phenomenon of inaccurate pronunciation caused by spelling errors. However, compared with traditional pronunciation prediction methods based on dictionaries, sequence-to-sequence models rely on larger data sets for effective training and generalization, because complex and parameter-rich neural network models require more training data to ensure the reliability and stability of their prediction performance. Different languages have their unique phonemes and pronunciation rules, and researchers need to design conversion systems that can handle these unique phenomena.

[0003] Laotian is a writing system derived from the Latin alphabet, and its phoneme-to-character mapping relationship is complex and influenced by multiple languages and dialects. Unique phonemes in Laotian such as and , etc. need to be accurately converted. Therefore, developing an efficient and accurate Laotian character-to-phoneme system is of great significance for improving language technology applications in the region. SUMMARY

[0004] The problem to be solved by the present application is: the present application provides a Laotian character-to-phoneme method based on transfer learning, aiming to utilize pre-trained models on other languages and fine-tune them on limited Laotian data to achieve efficient phoneme conversion. This method not only alleviates the problem of data scarcity, but also speeds up the training process of the model and improves the generalization ability of the model on new tasks.

[0005] This invention constructs a more robust Lao character-to-phoneme model through transfer learning, laying the foundation for subsequent speech synthesis. The Transformer architecture is a successful general sequence-to-sequence model designed specifically for neural machine translation. Since G2P can be viewed as a translation problem, translating characters into their phonetic counterparts, the Transformer architecture can be applied to G2P problems. Because Thai, which has high linguistic similarity to Lao, is also a low-resource language and cannot be used as a pre-training language, this invention chooses Chinese as the pre-training language. Based on the Transformer architecture, a Chinese G2P model is first trained using an open-source Chinese dataset. The decoding parameters of this model are then initialized into a student model and fine-tuned using a Lao dataset of 10,000 characters to improve the accuracy of the Lao character-to-phoneme model.

[0006] The technical solution of this invention is: a method for Lao character-to-phoneme conversion through transfer learning, comprising:

[0007] Step 1: Data selection and preprocessing; Select the CPP Chinese-Lao self-constructed dataset;

[0008] Step 2: Capture the semantic relationships and contextual information between characters using the Transformer encoder module;

[0009] Step 3: Fine-tuning the Transformer model using the fine-tuning module: A pre-trained Chinese Transformer model was introduced, and the embedding layer and output fully connected layer of the decoder were replaced to adapt to different output phoneme set size requirements; finally, the adjusted model was fine-tuned under the same language and training dictionary size conditions; the fine-tuned Transformer model was used to convert Lao characters to phonemes.

[0010] As a further aspect of the present invention, in Step 1, the number of words in the dictionary used to generate training data is limited; when testing the model, all words in the dictionary are used to generate test data, and sentences containing words not included in the dictionary are excluded; in addition, the frequency of each word in the training set is statistically analyzed, and based on this frequency information, it is determined which words to add to the mini-dictionary.

[0011] As a further aspect of the present invention, Step 2 includes:

[0012] The BERT pre-training language model with the character granularity of the Lao language is used as part of the Transformer encoder module; the input character sequence is first processed by a tokenizer, including text conversion to lowercase, removal of punctuation, and tokenization operation; the BERT adopts the WordPiece tokenization method, and the word is further divided into subwords to optimize the size of the vocabulary and improve the generalization ability of the model; by introducing the BERT pre-training language model, the character-level information is fully utilized to enhance the ability to cope with the challenge of lack of data in the Lao language character-to-phoneme task; the feature extraction and representation learning ability of the BERT model is integrated into the character-to-phoneme model to better capture the semantic relationship and context information between characters.

[0013] As a further scheme of the present application, the Step3 comprises:

[0014] The pre-trained Transformer model is fine-tuned; during the training of the Transformer model, the consistency of all model hyperparameters is ensured to ensure the reliability of the experiment; the Transformer model used contains 4.1M parameters in the training process, and the model state with the optimal validation loss is saved after 100 epoch iterations of training on the Chinese CPP dataset; then, the Transformer model is trained from the initial state on the mixed dataset combining Chinese and Lao language; the size range of the dictionary is adjusted, and after 50 epoch training, the model with the minimum loss on the validation set is selected for further evaluation work.

[0015] As a further scheme of the present application, the Step3 comprises:

[0016] The pre-trained Chinese Transformer model is introduced, and the embedding layer and output fully connected layer of the decoder part are replaced to adapt to the different output phoneme set size requirements; finally, the adjusted model is fine-tuned under the same language and training dictionary size conditions, and the model with the most excellent validation loss at the end of training is selected again for subsequent analysis and application.

[0017] The present application has the following beneficial effects:

[0018] 1. The present application proposes a Lao language character-to-phoneme method based on transfer learning, which aims to use the pre-trained model on other languages and fine-tune the limited Lao language data to realize efficient phoneme conversion. This method not only can alleviate the problem of data scarcity, but also can accelerate the training process of the model and improve the generalization ability of the model on new tasks;

[0019] 2、Through the migration learning, a more robust Lao character to phoneme model can be constructed, and a foundation is laid for subsequent speech synthesis. The Transformer architecture is a successful general sequence-to-sequence model designed for neural machine translation. Since G2P can be regarded as a translation problem, translating graphemes into their phonetic counterparts, the Transformer architecture can be applied to the G2P problem. Since Thai, which has a high language similarity with Lao, is also a low-resource language and cannot be used as a pre-training language, the present application selects Chinese as a pre-training language. Based on the Transformer architecture, a Chinese G2P model is first trained using a Chinese open-source dataset, the decoding end parameters of the model are initialized to the student model, and a Lao dataset with a size of ten thousand is used for fine-tuning training, so as to improve the accuracy of the Lao character to phoneme model;

[0020] 3、The present application realizes the significant improvement of the model performance by pre-training the Transformer model on the Chinese dataset and then fine-tuning using the Lao dataset. Compared with the model trained only using the Lao dataset, the fine-tuned model reduces the word error rate (WER) by 13.25% and the phoneme error rate (PER) by 3.15%, and the final word error rate is 21.67% and the phoneme error rate is 4.51%. The experimental results verify the effectiveness of the pre-training combined with the fine-tuning strategy in the character to phoneme task in the low-resource scenario. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a general design block diagram of a Lao character to phoneme method based on migration learning proposed by the present application;

[0022] Figure 2 is a Lao speech synthesis character to phoneme encoder diagram proposed by the present application; DETAILED DESCRIPTION

[0023] The Lao character to phoneme method based on migration learning proposed by the present application will be further described below in combination with the drawings. It should be noted that the drawings are very simplified and are only used to facilitate and clarify the purpose of assisting in the description of the embodiments of the present application.

[0024] Embodiment 1: As shown in the figure, a Lao character to phoneme method based on migration learning includes: Figure 1

[0025] ​Step 1: Data selection and preprocessing: Selecting the CPP Chinese-Lao self-constructed dataset; CPP is an open-source Chinese multi-phonetic character dataset. In order to conduct experiments, the CPP Chinese-Lao self-constructed dataset is selected as the training and testing data for the character-to-phoneme model. In order to make the language model more suitable for actual application scenarios, the number of words in the dictionary used to generate training data is limited. When testing the model, all words in the dictionary are used to generate test data, and sentences containing words not included in the dictionary are excluded. In addition, in order to ensure that even a small subset selected from a large dictionary contains some high-frequency common words, the frequency of each word in the training set is counted. According to this frequency information, it is decided which words to add to the small dictionary;

[0026] Step 2: Capture semantic relationships and contextual information between characters through the Transformer encoder module; the Step 2 includes:

[0027] As we focus on the character-to-phoneme task under scarce resources, in order to overcome the problem of data scarcity in the Lao character-to-phoneme task, we introduce more text features to assist in the prediction of phoneme sequences. We use the Lao character-level BERT pre-training language model as part of the Transformer encoder module; the input character sequence is first processed by the tokenizer, which includes converting text to lowercase, removing punctuation, and performing tokenization. BERT uses the WordPiece tokenization method, which further divides words into subwords to optimize the size of the vocabulary and improve the generalization ability of the model. As shown in Figure 2 by introducing the BERT pre-training language model, we aim to make full use of character-level information to enhance our ability to cope with the challenges of data scarcity in the Lao character-to-phoneme task; the feature extraction and representation learning capabilities of the BERT model are integrated into the character-to-phoneme model to better capture semantic relationships and contextual information between characters.

[0028] Step 3: Fine-tuning the Transformer model through the fine-tuning module: A pre-trained Chinese Transformer model is introduced, and the embedding layer and output fully connected layer of the decoder are replaced to adapt to different output phoneme set size requirements; finally, the adjusted model is fine-tuned under the same language and training dictionary size conditions; the fine-tuned Transformer model is used for Lao character-to-phoneme. The Step 3 includes:

[0029] In order to make the pre-trained Transformer model better apply to the Lao character-to-phoneme task, the pre-trained Transformer model is fine-tuned; during the training of the Transformer model, the consistency of all model hyperparameters is ensured to ensure the reliability of the experiment; the Transformer model used in the training process contains a total of 4.1M parameters, through the pre-training of the Chinese CPP data set, after 100 epoch iterative training, the model state with the most ideal validation loss is saved; then, starting from the initial state, the Transformer model is trained on the mixed data set combining Chinese and Lao; in order to adapt to the needs of different dictionary sizes, the size range of the dictionary is adjusted, including [1K, 3K, 5K, 10K] words. After 50 epoch training, the model with the smallest loss on the validation set is selected for further evaluation work.

[0030] The pre-trained Chinese Transformer model is introduced, and the embedding layer and output fully connected layer of the decoder part are replaced to adapt to the size requirements of different output phoneme sets; finally, the adjusted model is fine-tuned for 25 epochs under the same language and training dictionary size conditions, and the model with the most excellent validation loss at the end of training is selected again for subsequent analysis and application.

[0031] The present application designs a comparative experiment with other schemes to prove the effectiveness of the proposed method. First, the Transformer-based model is used to train independently on the self-constructed Lao character-to-phoneme data set. In order to evaluate the performance of the model in the low-resource language environment and its potential improvement space, a series of comparative experiments are designed. In particular, the independently trained Transformer model is compared and analyzed with the model preprocessed by transfer learning to explore the gain effect of fine-tuning the pre-trained Transformer model on the character-to-phoneme task in the low-resource language environment.

[0032] In addition, in order to comprehensively evaluate the performance of the proposed method, it is also compared with traditional baseline models such as recurrent neural network (RNN) and bidirectional long short-term memory (BiLSTM). These models are selected as baseline models because they have been widely studied and used in sequence processing tasks. Through this comparison, the advantages and potential limitations of the Transformer structure in processing character-to-phoneme tasks compared with other mainstream models are revealed. The experimental results are shown in Table 1.

[0033] Table 1: WER and PER performance of different models, here the Chinese dataset is the whole CPP dataset

[0034]

[0035]

[0036] From the experimental data, it can be observed that in all models, when the training data is scarce, whether it is RNN, BiLSTM or Transformer, their performance is not satisfactory. However, in the case of sufficient data sets, for the Chinese character to phoneme task, each model can achieve a relatively ideal performance.

[0037] It is particularly noteworthy that by pre-training the Transformer model on the Chinese dataset and then fine-tuning it with the Lao dataset, the resulting character-to-phoneme model shows a significant performance improvement. Although its performance did not surpass the Chinese character-to-phoneme model, compared to the model trained only on the Lao dataset, this model achieved a significant 13.25% reduction in word error rate (WER) and a 3.15% reduction in phoneme error rate (PER). This result reflects the effectiveness of the pre-training plus fine-tuning strategy in low-resource language processing.

[0038] To test whether the proposed transfer learning method can bring performance improvement to the character-to-phoneme task of different languages, experiments were conducted on different languages in the same way. Due to the high cost of obtaining low-resource languages, experiments were conducted on 10,000 data of English, Japanese, Korean, and German datasets, which are the same size as the Lao dataset. This can also be considered as a low-resource language at the dataset level. The experimental results are shown in Table 2:

[0039] Table 2: Performance of the proposed method on different language datasets, each language dataset has 10,000 texts, w / indicates fine-tuning training with Transformer pre-trained on Chinese dataset,

[0040] w / o indicates direct training on Transformer

[0041]

[0042]

[0043] In the analysis of Table 2, it is found that using the same size of data set, including English, Japanese, Korean, German and Lao, there are significant differences in the performance obtained when using the transformer model for training. In particular, Lao is significantly lagging behind in performance, which may be due to its unique writing system and the complexity of its pronunciation characteristics, which makes it difficult for the model to capture its regularity, and makes it more difficult to model compared with other mainstream languages. From this, a preliminary conclusion can be drawn: the existing general character-to-phoneme method is not suitable for direct application to Lao.

[0044] Further, when using the proposed method, that is, using a pre-trained Chinese Transformer model for training, it is found that the performance of each mainstream language has been improved. By analyzing the WER (Word Error Rate) and PER (Phoneme Error Rate) indicators in detail, it is observed that compared with English and German, the performance of Japanese and Korean character-to-phoneme models has improved more significantly. This result may be due to historical and cultural ties, as Chinese, Japanese and Korean have many commonalities in language and writing. Since Japanese and Korean contain a large number of Chinese character loanwords, the pre-trained model obtained using the Chinese data set can provide more useful information when performing Japanese and Korean transfer learning, thereby more effectively promoting the learning and optimization of the model. Based on this assumption, the English data set is used to pre-train the Transformer model and fine-tune it on the Lao data set. The performance of the Chinese pre-trained model after fine-tuning on the Lao character-to-phoneme task is compared, and the same size of German data set as the Lao data set is used to compare the performance of the pre-trained Chinese Transformer model and the pre-trained English Transformer model after fine-tuning on the German data set. To ensure the objectivity of the experimental results, the size of the English data set used is consistent with the size of the Chinese data set, and the experimental results are shown in Table 3.

[0045] Table 3 verifies the influence of language commonality on character-to-phoneme models

[0046]

[0047] According to linguistic conclusions, English and German belong to the West Germanic branch of the Germanic branch of the Indo-European language family, and have high language similarity. By analyzing the experimental results shown in Table 2.4, it is found that when a small amount of German data set is used to fine-tune the pre-trained English Transformer model, the performance of the German character-to-phoneme model is significantly improved, the WER value is reduced to 18.74%, and the PER value is only 3.95%. In contrast, fine-tuning the English pre-trained Transformer model with the Lao data set and fine-tuning the Chinese pre-trained Transformer model with the German data set can improve the performance of the model to some extent, but it does not achieve the significant performance improvement that language similarity may bring. This result confirms the method proposed in this study: selecting a Chinese data set as a pre-training data set has a more significant effect on improving the performance of the Lao character-to-phoneme model.

[0048] The specific embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. A Lao character-to-phoneme method based on transfer learning, characterized in that: include: Step 1: Data selection and preprocessing; Select the CPP Chinese-Lao self-constructed dataset; Step 2: Capture the semantic relationships and contextual information between characters using the Transformer encoder module; Step 3: Fine-tuning the Transformer model using the fine-tuning module: A pre-trained Chinese Transformer model was introduced, and the embedding layer and output fully connected layer of the decoder were replaced to adapt to different output phoneme set size requirements; finally, the adjusted model was fine-tuned under the same language and training dictionary size conditions. Lao character to phoneme conversion using a finely tuned Transformer model; Step 2 includes: A Lao character-level BERT pre-trained language model is used as part of the Transformer encoder module; The input character sequence is first processed by a word segmenter, which includes converting the text to lowercase, removing punctuation marks, and performing word segmentation. BERT uses the WordPiece segmentation method to further subdivide words into sub-words to optimize the size of the vocabulary and improve the model's generalization ability. By introducing the BERT pre-trained language model, it aims to make full use of character-level information to enhance the ability to cope with the challenge of lacking data in the Lao character-to-phoneme task. The feature extraction and representation learning capabilities of the BERT model are integrated into the character-to-phoneme model to better capture the semantic relationships and contextual information between characters.

2. The Lao character-to-phoneme method based on transfer learning according to claim 1, characterized in that: In Step 1, the number of words in the dictionary used to generate training data is limited; when testing the model, all words in the dictionary are used to generate test data, and sentences containing words not included in the dictionary are excluded; in addition, the frequency of each word in the training set is statistically analyzed, and based on this frequency information, it is decided which words to add to the mini-dictionary.

3. The Lao character-to-phoneme method based on transfer learning according to claim 1, characterized in that: Step 3 includes: The pre-trained Transformer model was fine-tuned. During the Transformer model training process, the hyperparameters of all models were kept consistent to ensure the reliability of the experiment. The Transformer model used contained a total of 4.1M parameters during training. It was pre-trained on the Chinese CPP dataset and iterated for 100 epochs to save the model state with the ideal validation loss. Subsequently, the Transformer model was trained from the initial state on a mixed dataset combining Chinese and Lao. The size range of the dictionary was adjusted, and after 50 epochs of training, the model with the smallest loss on the validation set was selected for further evaluation.

4. The Lao character-to-phoneme method based on transfer learning according to claim 1, characterized in that: Step 3 includes: A pre-trained Chinese Transformer model was introduced, and the embedding layer and output fully connected layer of the decoder were replaced to adapt to different output phoneme set size requirements. Finally, the adjusted model was fine-tuned under the same language and training dictionary size conditions, and the model with the best validation loss at the end of training was selected for subsequent analysis and application.

Images