Lao-Chinese neural machine translation method and device based on fusion of efficient parameter migration

Abstract

The present invention relates to a Lao-Chinese neural machine translation method and device integrating efficient parameter migration, and belongs to the technical field of natural language processing. In order to solve the problem that multiple characters are transcribed into the same Romanized character during the romanization process, resulting in loss of semantic information, which affects the migration effect from Thai to Lao, the present invention proposes a Lao-Chinese neural machine translation method integrating efficient parameter migration, including text data preprocessing, Romanization information fusion, Lao-Chinese neural machine translation model training integrating efficient parameter migration, and Lao-Chinese neural machine translation. According to the functional modularization of these four parts, an Lao-Chinese neural machine translation device integrating efficient parameter migration is made, and the input Lao sentences are translated. The present invention effectively improves the translation effect from Lao to Chinese.

Description

Technical Field

[0001] The present invention relates to an Old-Chinese neural machine translation method and device based on fusion of efficient parameter migration, belonging to the technical field of natural language processing. Background Art

[0002] The word embedding transfer in transfer learning mainly improves the model transfer effect by improving the vocabulary alignment between the parent and child languages. Thai and Lao are difficult to align due to inconsistent character encoding. Romanization can effectively improve the similarity between Thai and Lao texts, but we found that the semantics of the original text is lost during the romanization process. Specifically, we found that there are multiple characters romanized into the same character during the romanization transcription process. Since many characters in Thai and Lao have the same pronunciation, their romanized representations are also the same. This situation where multiple characters are romanized into the same character will cause the problem of semantic loss after the romanization transcription of Thai and Lao texts, making it impossible for the model to effectively learn the language knowledge of Thai and Lao, thus affecting the model translation performance. Summary of the invention

[0003] In view of the above problems, the present invention provides a Lao-Chinese neural machine translation method and device based on fusion of efficient parameter migration, which is used to improve the translation performance from Lao to Chinese.

[0004] The technical solution of the present invention is: in the first aspect, the present invention provides an Old-Chinese neural machine translation method based on fusion of efficient parameter migration, the method comprising:

[0005] Step 1: Text data preprocessing: construct Thai-Chinese and Lao-Chinese bilingual parallel corpora, and use Thai and Lao romanization transcription tools to transcribe Thai and Lao into corresponding romanized text character representations;

[0006] Step 2: Romanized information fusion: Use the original text information to supplement the information and fuse the original text information with the Romanized text character representation;

[0007] Step 3: Training of Lao-Chinese neural machine translation model integrating efficient parameter migration and Lao-Chinese neural machine translation: When migrating the Thai-Chinese translation model to the Lao-Chinese translation model, add an adapter to adjust the model, reduce the training parameters of the model training process, save the trained model, and deploy it to the server to achieve Lao text translation into Chinese.

[0008] As a further solution of the present invention, step 1 comprises:

[0009] Step 1.1, obtain Thai-Chinese, Lao-Chinese and Thai-Lao bilingual parallel corpora from OPUS and Asian language treebanks, and obtain Thai-Chinese comparable corpora from Thai-Chinese bilingual websites; clean the comparable corpora by web page tag removal, regular matching and length screening, calculate the similarity of sentences based on pronunciation similarity, screen out Thai-Chinese and Lao-Chinese pseudo-parallel corpora with high matching, and manually annotate them to obtain Thai-Chinese and Lao-Chinese parallel corpora as experimental data sets, and divide them into training set, validation set and test set;

[0010] Step 1.2: Use Thai and Lao romanization tools to romanize the Thai-Chinese and Lao-Chinese parallel corpora in step 1.1 to obtain romanized Thai-Chinese and Lao-Chinese parallel corpora.

[0011] As a further solution of the present invention, step 2 comprises:

[0012] Step 2.1: Additional Thai and Lao original texts are introduced to supplement the text semantic information missing from the romanized transcription;

[0013] Step 2.2: Vectorize the original text and the romanized text respectively, and use the cross-attention mechanism to make the vector of the romanized text pay attention to the language information of the original text and supplement the semantic information lost in the romanization transcription process; for the original text and Romanized text , where m and n are the length of the romanized text and the length of the original text respectively; the original text x and the romanized text x_rom are sent to the embedding layer Embedding to get the original text vector representation and Romanized text vector representation , as shown in the following formula;

[0014] ;

[0015] ;

[0016] Step 2.3: Use the cross attention mechanism and self-attention mechanism to represent the original text vector and Romanized text vector representation Processing is performed to obtain features that integrate the original text information and Romanized text features ;

[0017] Step 2.4: Features of fused original text information and Romanized text features The final feature representation H is obtained by weighted fusion, where α is a hyperparameter, as shown in the following formula:

[0018] .

[0019] As a further solution of the present invention, in step 2.3, the feature of the fusion of the original text information is calculated. When , the Romanized text vector representation is the query vector, the original text vector representation are key vectors and value vectors, and the features of the original text information are obtained through the cross attention mechanism. , as shown below:

[0020] .

[0021] As a further solution of the present invention, in step 2.3, the Romanized text features are calculated When using the self-attention mechanism, the Romanized text vector representation As query vector, key vector and value vector respectively, the romanized text features are obtained through the self-attention layer , as shown below:

[0022] .

[0023] As a further solution of the present invention, step 3 comprises:

[0024] Step 3.1, training Thai-Chinese neural machine translation model: using Thai-Chinese corpus and Thai romanized text to train Thai-Chinese neural machine translation model as the basic model; during training, the Thai original text and Thai romanized text are fused through the cross attention mechanism;

[0025] Step 3.2, model migration: During the model migration phase, the Thai romanized text representation is migrated; for the original text representation, the word frequency correlation between Thai and Lao is used for migration, and the subword embeddings of Thai and Lao are migrated in the order of word frequency;

[0026] Step 3.3, model migration fine-tuning: migrate the trained Thai-Chinese neural machine translation model to the Lao-Chinese neural machine translation model, and use the romanized Lao-Chinese parallel corpus in step 3.1 to fine-tune the adapter of the migrated Lao-Chinese neural machine translation model;

[0027] Step 3.4: Save the model trained in step 3.3 in ".pt" format, load the model into memory through the Flask framework; use the Flask framework to deploy the model as an API interface; on the Web side, call the Lao-Chinese neural machine translation model deployed to the server side to translate the input Lao language, and then obtain the corresponding Chinese translation effect of Lao language.

[0028] In a second aspect, the present invention also provides an Old-Chinese neural machine translation device based on fusion of efficient parameter migration, which includes a module for executing the method of the first aspect.

[0029] The beneficial effects of the present invention are:

[0030] 1. In order to alleviate the problem of semantic loss caused by romanization of multiple characters into the same character in the romanization transcription process, the present invention introduces original text information to supplement the lost romanization information;

[0031] 2. The present invention integrates the Lao-Han neural machine translation method with efficient parameter migration, and adjusts the migrated model by introducing an adapter, thereby reducing the training parameters in the model training process and improving the model training efficiency;

[0032] 3. The present invention translates input Lao sentences, and the present invention effectively improves the translation effect from Lao to Chinese. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a network architecture diagram of the Lao-Han neural machine translation method that integrates efficient parameter migration in the present invention. DETAILED DESCRIPTION

[0034] Example 1: Figure 1 As shown, the Lao-Han neural machine translation method based on the fusion of efficient parameter migration, the specific steps of the method are as follows:

[0035] Step 1: construct Thai-Chinese and Lao-Chinese bilingual parallel corpora, and use Thai and Lao romanization transcription tools to transcribe Thai and Lao into corresponding romanized text character representations;

[0036] Step 1.1, obtain Thai-Chinese, Lao-Chinese and Thai-Lao bilingual parallel corpora from OPUS and Asian language treebanks, and obtain Thai-Chinese comparable corpora from Thai-Chinese bilingual websites; clean the comparable corpora by web page tag removal, regular matching and length screening, calculate the similarity of sentences based on pronunciation similarity, screen out Thai-Chinese and Lao-Chinese pseudo-parallel corpora with high matching, and manually annotate them to obtain Thai-Chinese and Lao-Chinese parallel corpora as experimental data sets, and divide them into training set, validation set and test set;

[0037] Step 1.2: To address the problem of semantic information loss caused by multiple characters being transcribed into the same character during the romanization process of Thai and Lao languages, the Thai-Chinese and Lao-Chinese parallel corpora in step 1.1 are romanized using Thai and Lao romanization tools to obtain romanized Thai-Chinese and Lao-Chinese parallel corpora.

[0038] Step 2: Use the original text information to supplement the information and fuse the original text information with the romanized text character representation; build a romanized information fusion module. The romanized information fusion module effectively supplements the lost romanized information and improves the model's ability to represent text;

[0039] Step 2.1: In view of the situation that multiple characters are romanized to the same character in the text transcription process of the existing Thai and Lao romanization tools, since many characters in Thai and Lao have the same pronunciation, their romanized representations are also the same, the original Thai and Lao texts are additionally introduced to supplement the text semantic information missing in the romanized transcription;

[0040] Step 2.2: In order to combine the original Thai and Lao texts with the romanized texts, the original texts and the romanized texts are vectorized respectively. By using the cross-attention mechanism, the vector of the romanized text can focus on the language information of the original text and supplement the semantic information lost in the romanized transcription process. and Romanized text , where m and n are the length of the romanized text and the length of the original text respectively; the original text x and the romanized text x_rom are sent to the embedding layer Embedding to get the original text vector representation and Romanized text vector representation , as shown in the following formula;

[0041] ;

[0042] ;

[0043] Step 2.3: Use the cross attention mechanism and self-attention mechanism to represent the original text vector and Romanized text vector representation Processing is performed to obtain features that integrate the original text information and Romanized text features ;

[0044] In step 2.3, the features of the fused original text information are calculated When , the Romanized text vector representation is the query vector, the original text vector representation are key vectors and value vectors, and the features of the original text information are obtained through the cross attention mechanism. , as shown below:

[0045] .

[0046] In step 2.3, the Romanized text features are calculated When using the self-attention mechanism, the Romanized text vector representation As query vector, key vector and value vector respectively, the romanized text features are obtained through the self-attention layer , as shown below:

[0047] .

[0048] Step 2.4: Features of fused original text information and Romanized text features The final feature representation H is obtained by weighted fusion, where α is a hyperparameter, as shown in the following formula:

[0049] .

[0050] Step 3: To address the problem of low model training efficiency due to the large number of fine-tuning parameters during the migration process, an adapter is added to adjust the model during the migration of the Thai-Chinese translation model to the Lao-Chinese translation model, reduce the training parameters of the model training process, save the trained model, and deploy it to the server to achieve Lao text translation into Chinese.

[0051] Step 3.1, training Thai-Chinese neural machine translation model: using Thai-Chinese corpus and Thai romanized text to train Thai-Chinese neural machine translation model as the basic model; during training, the original Thai text and the Thai romanized text are fused through the cross-attention mechanism; the original text information is used to alleviate the problem of text information loss in the romanization transcription process;

[0052] Step 3.2, model migration: During the model migration phase, migrate the Thai romanized text representation part so that the word embeddings of the same part of the romanized representation can be effectively migrated to the Lao romanized word embeddings; for the original text representation part, use the word frequency correlation between Thai and Lao to migrate the Thai and Lao subword embeddings in the order of word frequency, so that the subword embeddings with correlation can be effectively aligned, and the subword embeddings with the same meaning can be correctly embedded to improve the word embedding migration effect;

[0053] Step 3.3, model migration and fine-tuning: Migrate the trained Thai-Chinese neural machine translation model to the Lao-Chinese neural machine translation model. Since the model is not trained for Lao, use the romanized Lao-Chinese parallel corpus in step 3.1 to fine-tune the adapter of the migrated Lao-Chinese neural machine translation model.

[0054] Step 3.4: Save the model trained in step 3.3 in ".pt" format, and load the model into memory through the Flask framework; avoid frequent model loading caused by request results and improve the running speed of the translation model; use the Flask framework to deploy the model as an API interface; on the Web side, call the Lao-Chinese neural machine translation model deployed to the server side to translate the input Lao language, and then obtain the corresponding Chinese translation effect of Lao language.

[0055] According to the concept of the present invention, the present invention also provides an Old-Chinese neural machine translation device based on fusion of efficient parameter migration, which includes the following integrated modules:

[0056] Corpus construction and romanization transcription module: used to construct Thai-Chinese and Lao-Chinese bilingual parallel corpora, and transcribe Thai and Lao into corresponding romanized text character representations using Thai and Lao romanization transcription tools;

[0057] Romanized information fusion module: used to supplement information with original text information and fuse the original text information with Romanized text character representation;

[0058] Translation module: used to add an adapter to adjust the model when migrating the Thai-Chinese translation model to the Lao-Chinese translation model, reduce the training parameters in the model training process, save the trained model, deploy it to the server, and translate Lao text into Chinese.

[0059] In order to verify the effect of the Lao-Han neural machine translation method integrating efficient parameter migration proposed in the present invention, the following experiments were designed for analysis.

[0060] In order to illustrate the effect of the present invention, the present invention conducted the following experiment, and the experimental data used are as follows. The Thai-Chinese data of the present invention mainly includes corpora collected from websites such as the Asian Language Treebank, corpora collected through bilingual websites on the Internet, and manually constructed parallel corpora, totaling 254,741 items. The details are shown in Table 1.

[0061] Table 1 is the data set

[0062]

[0063] The neural network model of the experiment is implemented in Torch, the Torch version is 1.8, the compilation language is Python3.8, and the experiment is conducted on a single NVIDIA 3090 GPU. The experiment uses the Jieba word segmentation tool to segment Chinese words. BPE is applied to subword segmentation of the source language and the target language, and the word list size is 16k. The experiment selects the Transformer model as the basic model, and the encoder and decoder of the model are set to 6 layers respectively. The dimension of the word vector in the encoder and decoder is set to 512 dimensions. The optimizer selects the Adam optimizer with parameters set to β1= 0.9 and β2= 0.98 to optimize the model. The experiment uses the warm-up strategy with warm steps s = 4000 to adjust the learning rate, and each batch contains about 4096 words. The model is trained until the BLEU value of the validation set does not improve for 10 consecutive times, then the model is considered to converge and training is stopped. During the decoding process, the value of the heuristic search algorithm during decoding is set to 5. For evaluation, the experiment uses the standard BLEU scoring standard to detect the performance of the model.

[0064] Experiment 1,In order to verify the effectiveness of the proposed method, the following experiment is set up, and the experimental results are shown in Table 2.

[0065] Table 2 shows the similarity comparison between Thai and Lao sentences.

[0066]

[0067] As can be seen from Table 2, the following results can be obtained: the lowest BLEU value of 3.33 was obtained by using only Transformer to train the Lao-Chinese model, that is, only using a small amount of Lao Chinese data for training cannot effectively learn language knowledge, resulting in poor translation performance. In comparison, the model using the Token Matching transfer learning method achieved a BLEU value of 7.38, which is significantly better than the model using only the Transformer, verifying that transfer learning can effectively transfer Thai language knowledge to Lao, thereby enhancing the model's ability to learn Lao. The effect of using the romanized representation of the original text for transfer based on Token Matching is better, with a BLEU value of 9.83. This is because the romanization of Thai and Lao improves the text similarity between Thai and Lao, and can better align the Thai vocabulary with the Lao vocabulary, improving the transfer effect. The use of Multi-Encoder Architectures to fuse the original text and the romanized text achieved a BLEU value of 10.15, which is better than the translation effect of using only romanized text. This shows that the fusion of the original text and the romanized text through Multi-Encoder Architectures can effectively supplement the semantic information lost in the romanization transcription process. However, the use of Straight Concatenation to fuse the original text and the romanized text resulted in a decrease in the translation performance of the model, with a BLEU value of only 5.16. This shows that after Straight Concatenation processing, the model input sequence will be too long, making it difficult for the model to learn language knowledge, thereby causing the model's translation performance to decrease. The method proposed in the present invention achieved the best effect in the translation direction of Lao-Chinese, with a BLEU value of 10.67, indicating that the fusion of the original text and the romanized text by cross-attention can effectively supplement the original text information.

[0068] Experiment 2: In order to verify the effectiveness of the proposed fusion of original text information and the introduction of adapters for model parameter adjustment in improving the old-Chinese neural machine translation, the present invention sets up an ablation experiment for comparison, and the experimental results are shown in Table 3. By analyzing the results of Table 3, the following results can be obtained: the BLEU in the direction of old-Chinese machine translation using the original text combined with romanization (i.e., the method of "the present invention method-adapter" in Table 3, that is, the method of subtracting the adapter from the method of the present invention) reaches 10.50, which is higher than the BLEU value of the model training using only romanized text. This shows that combining the original text with the romanized representation can effectively supplement the semantic information lost in the romanization process and improve the performance of the old-Chinese neural machine translation. Compared with the method of fine-tuning all parameters, the method of introducing an adapter (i.e., the "method of the present invention" in Table 3) for parameter adjustment achieved a BLEU value of 10.67, which was 0.17 BLEU higher than the method of "method of the present invention-adapter". This shows that using the adapter for parameter adjustment can effectively retain the language information corresponding to the romanized part and improve the model performance of the Old-Chinese neural machine translation.

[0069] Table 3 shows the translation effect of Lao

[0070]

[0071] Taking the translation result of Lao-Chinese as an example, the influence of the proposed method on the quality of the translation generated by the model is analyzed. As shown in Table 4, in the direction of Lao-Chinese translation, the baseline method translates "International News Weekly" into "News Weekly", while the proposed method can translate correctly. This shows that the proposed method can effectively translate Lao.

[0072] Table 4 shows examples of Lao-Chinese translation

[0073]

[0074] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An Old-Chinese Neural Machine Translation Method Based on Fusion of Efficient Parameter Transfer, Features: The method comprises: Step 1: Text data preprocessing: construct Thai-Chinese and Lao-Chinese bilingual parallel corpora, and use Thai and Lao romanization transcription tools to transcribe Thai and Lao into corresponding romanized text character representations; Step 2: Romanized information fusion: Use the original text information to supplement the information and fuse the original text information with the Romanized text character representation; Step 3: Lao-Chinese neural machine translation model training and Lao-Chinese neural machine translation integrating efficient parameter migration: add an adapter to adjust the model during the migration of the Thai-Chinese translation model to the Lao-Chinese translation model, reduce the training parameters of the model training process, save the trained model, deploy it to the server, and realize the translation of Lao text into Chinese; The step 1 comprises: Step 1.1, obtain Thai-Chinese, Lao-Chinese and Thai-Lao bilingual parallel corpora from OPUS and Asian language treebanks, and obtain Thai-Chinese comparable corpora from Thai-Chinese bilingual websites; clean the comparable corpora by web page tag removal, regular matching and length screening, calculate the similarity of sentences based on pronunciation similarity, screen out Thai-Chinese and Lao-Chinese pseudo-parallel corpora with high matching, and manually annotate them to obtain Thai-Chinese and Lao-Chinese parallel corpora as experimental data sets, and divide them into training set, validation set and test set; Step 1.2: Use Thai and Lao romanization tools to romanize the Thai-Chinese and Lao-Chinese parallel corpora in step 1.1 to obtain romanized Thai-Chinese and Lao-Chinese parallel corpora; The step 2 comprises: Step 2.1: Additional Thai and Lao original texts are introduced to supplement the text semantic information missing from the romanized transcription; Step 2.2: Vectorize the original text and the romanized text respectively, and use the cross-attention mechanism to make the vector of the romanized text pay attention to the language information of the original text and supplement the semantic information lost in the romanization transcription process; for the original text and Romanized text , where m and n are the length of the romanized text and the length of the original text respectively; the original text x and the romanized text x_rom are sent to the embedding layer Embedding to get the original text vector representation and Romanized text vector representation , as shown in the following formula; ； ； Step 2.3: Use the cross attention mechanism and self-attention mechanism to represent the original text vector and Romanized text vector representation Processing is performed to obtain features that integrate the original text information and Romanized text features ; Step 2.4: Features of fused original text information and Romanized text features The final feature representation H is obtained by weighted fusion, where α is a hyperparameter, as shown in the following formula: ； The step 3 comprises: Step 3.1, training Thai-Chinese neural machine translation model: using Thai-Chinese corpus and Thai romanized text to train Thai-Chinese neural machine translation model as the basic model; during training, the Thai original text and Thai romanized text are fused through the cross attention mechanism; Step 3.2, model migration: During the model migration phase, the Thai romanized text representation is migrated; for the original text representation, the word frequency correlation between Thai and Lao is used for migration, and the subword embeddings of Thai and Lao are migrated in the order of word frequency; Step 3.3, model migration fine-tuning: migrate the trained Thai-Chinese neural machine translation model to the Lao-Chinese neural machine translation model, and use the romanized Lao-Chinese parallel corpus in step 3.1 to fine-tune the adapter of the migrated Lao-Chinese neural machine translation model; Step 3.4: Save the model trained in step 3.3 in ".pt" format, load the model into memory through the Flask framework; use the Flask framework to deploy the model as an API interface; on the Web side, call the Lao-Chinese neural machine translation model deployed to the server side to translate the input Lao language, and then obtain the corresponding Chinese translation effect of Lao language.

2. The Lao-Han neural machine translation method based on fusion of efficient parameter migration according to claim 1, Features: In step 2.3, the features of the fused original text information are calculated When , the Romanized text vector representation is the query vector, the original text vector representation are key vectors and value vectors, and the features of the original text information are obtained through the cross attention mechanism. , as shown below: 。 3. The Lao-Han neural machine translation method based on fusion of efficient parameter migration according to claim 1, Features: In step 2.3, the Romanized text features are calculated When using the self-attention mechanism, the Romanized text vector representation As query vector, key vector and value vector respectively, the romanized text features are obtained through the self-attention layer , as shown below: 。 4. Lao-Han neural machine translation device based on fusion of efficient parameter migration, It is characterized in that The method comprises a module for executing the method according to any one of claims 1 to 3.

Images