Method for constructing low-resource neural machine translation system based on pre-training knowledge enhancement

Abstract

The application discloses a low-resource neural machine translation system construction method based on pre-training knowledge enhancement, acquires a pre-training language model of a Transformer encoder on a network; acquires a low-resource language pair data set of a neural machine translation system to be constructed and cleanses; acquires a language analysis tool to analyze training data, masks a key word, and uses processed data to fine-tune the pre-training language model; builds a neural machine translation model, uses a Transformer decoder and randomly initializes parameters as a neural machine translation model decoder; gradually trains the encoder and the decoder of the neural machine translation model; deploys the trained neural machine translation model on a high-performance cloud server, builds a front end and a back end processing program, and completes the construction of the entire machine translation system. The application achieves performance improvement in multiple methods, and can further improve the performance of the machine translation model on the basis of two-stage training.

Description

Technical Field

[0001] This invention relates to a technique for enhancing low-resource neural machine translation, specifically a method for constructing a low-resource neural machine translation system based on pre-trained knowledge enhancement. Background Technology

[0002] Communication and exchange of information are essential to people's daily lives. Natural language, the medium of communication, is the most efficient information transmission medium developed by humankind over thousands of years. Its evolution has resulted in thousands of forms globally. As globalization permeates everyone's lives, the need for language translation is becoming increasingly urgent. In today's era of rapid advancements in computers and artificial intelligence, automatically converting between different languages ​​using computer software has become one of the most important goals of artificial intelligence. The technology used to achieve this goal is called machine translation (MT).

[0003] Due to the inherent complexity of machine translation tasks, large neural networks with complex structures and numerous parameters are typically required to achieve good translation results. This leads to two inherent drawbacks of neural networks: high demand for training data and high hardware resource requirements, which are particularly prominent in neural machine translation tasks. Specifically, for language pairs with abundant bilingual data, the training data can reach tens of millions or even hundreds of millions of data points. For these languages, training large neural machine translation models is perfectly sufficient. However, for minority languages, which account for 93% of all languages ​​in the world, the bilingual data is much smaller, perhaps only a few hundred thousand data points. Using such a small amount of data to train large neural machine translation models is prone to overfitting, and due to the limited data, the model struggles to learn sufficient linguistic knowledge for machine translation. Ultimately, the quality of machine translations generated from minority languages ​​is almost incomparable to that of major languages. This leads to the problem of low corpus resources in neural machine translation.

[0004] To address the issue of limited corpora in neural machine translation, current mainstream solutions include data augmentation and pre-training. The former involves expanding the training set size through various methods, such as back-translation and knowledge distillation. The latter utilizes cutting-edge pre-training techniques in natural language processing to compensate for the lack of knowledge in machine translation systems for less commonly taught languages ​​due to limited training data by using large pre-trained models that have already acquired general knowledge. Summary of the Invention

[0005] To address the shortcomings of existing technologies, such as low accuracy in low-resource language neural machine translation, the technical problem to be solved by this invention is to provide a method for constructing a low-resource neural machine translation system based on pre-trained knowledge enhancement, which can improve the accuracy of machine translation models.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] This invention provides a method for constructing a low-resource neural machine translation system based on pre-trained knowledge enhancement, comprising the following steps:

[0008] 1) Obtain a pre-trained language model trained on a large-scale corpus from the internet, with a Transformer encoder structure;

[0009] 2) Obtain the low-resource language pair dataset needed to build the neural machine translation system, and clean it to obtain the training data;

[0010] 3) Obtain language analysis tools that can identify keywords in text;

[0011] 4) Use language analysis tools to analyze the training data, mask the keywords, and fine-tune the pre-trained language model using the processed data;

[0012] 5) Build a neural machine translation model, including using a fine-tuned pre-trained language model as the encoder of the neural machine translation model, and using a Transformer decoder with randomly initialized parameters as the decoder of the neural machine translation model.

[0013] 6) Use a three-stage training method to gradually train the encoder and decoder of the neural machine translation model;

[0014] 7) Deploy the trained neural machine translation model on a high-performance cloud server and build the front-end and back-end processing programs to complete the construction of the entire machine translation system.

[0015] In step 2), the low-resource language pair dataset required by the user is obtained, and the dataset is cleaned using length ratio algorithm, HTML tag filtering, and special character filtering.

[0016] In step 3), obtain language analysis tools that can identify text keywords, specifically including: obtaining open-source stop word lists and syntax tree parsing tools from the internet, downloading and installing them.

[0017] In step 4), language analysis tools are used to analyze the training data, masking the keywords. Then, the pre-trained language model is fine-tuned using the processed data. The specific steps are as follows:

[0018] 401) For each sentence in the training corpus, use the stop word list to filter out words that are not stop words, and use the filtered words as the keyword set M;

[0019] 402) For each sentence in the training corpus, use a syntax tree parsing tool to analyze the dependency syntax tree of each text, and select the root node and the first-level child nodes of the syntax tree, and use the words corresponding to these nodes as the keyword set N;

[0020] 403) For each sentence in the training corpus, select words that belong to both keyword set M and keyword set N as the final keywords;

[0021] 404) For each sentence in the training corpus, calculate the proportion 'a' of the final filtered keywords to all words. If 'a' is less than K, then mask all keywords while randomly masking non-keywords to make the masking proportion reach K. If 'a' equals K, then mask all keywords. If 'a' is greater than K, then randomly select a portion of the keywords to make the masking proportion reach K.

[0022] 405) Fine-tune the pre-trained language model using masked corpus.

[0023] In step 6), the encoder and decoder of the neural machine translation are trained step by step using a three-stage training method, specifically as follows:

[0024] 601) First, train the decoder part of the neural machine translation model with randomly initialized parameters separately until convergence;

[0025] 602) Then simultaneously train the encoder of the neural machine translation model that already contains a large amount of language knowledge and the decoder that can be adapted to the pre-trained language model;

[0026] 603) Finally, train the neural machine translation model decoder separately.

[0027] The present invention has the following beneficial effects and advantages:

[0028] 1. This invention proposes a method for constructing a low-resource neural machine translation system based on pre-trained knowledge enhancement. The targeted and purposeful masking strategy is a starting point for tapping the potential of pre-trained models. It can improve the actual performance of pre-trained models using only 160,000 data points and achieve performance improvements across multiple methods. As an initial attempt, the core of this invention is very simple: finding keywords and performing targeted masking around those keywords. Subsequent work can continue along this line of thought.

[0029] 2. The three-stage training method proposed in this invention enables the decoder to more effectively utilize the knowledge of the pre-trained model. Based on the previous two-stage training method, this invention proposes a three-stage training method. The additional training stage can prevent the pre-trained model from overfitting further and make the decoder more compatible with the pre-trained model. Experiments show that this method can further improve the performance of the machine translation model on the basis of two-stage training and can surpass other excellent methods in the field. Attached Figure Description

[0030] Figure 1 This invention provides a method for fine-tuning masked keywords in the pre-trained model.

[0031] Figure 2 This is an example of the effect of using multiple language analysis tools to filter text keywords in this invention;

[0032] Figure 3 This invention presents a three-stage training method for the neural machine translation model. Detailed Implementation

[0033] This invention uses a pre-trained model knowledge-enhanced neural machine translation system as a technical approach. Based on this, it improves and proposes efficient fine-tuning methods and joint training methods to fully integrate pre-trained knowledge into the neural machine translation system. First, in reproducing the two-stage training method, this invention hypothesizes that the convergence state of the two stages may be the result of a dynamic balance between the negative impact of overfitting of the pre-trained model and the positive impact of the machine translation decoder continuously adapting to the pre-trained model. Therefore, a third stage of training is introduced, which allows the decoder to better adapt to the pre-trained model while preventing further overfitting on small datasets. Furthermore, this invention improves the masking language model method in the domain adaptation stage. The original masking language model adopts a random masking strategy, that is, randomly selecting a certain number of words from the input to mask them, and training the model to predict these masked words. However, in natural language, some words have low information content, while others have high information content. This invention hypothesizes that if the model can learn deeply about these high-information keywords, it can improve the model's understanding of the text. Therefore, this invention designs a method for discovering keywords in machine translation data, masking keywords, and predicting keywords. It improves the original masked language model in both the masking and prediction stages, which not only enables the pre-trained model to adapt to the domain, but also allows the pre-trained model to learn more language knowledge and enhance its language representation ability.

[0034] The technical solution adopted in this invention is:

[0035] 1) Obtain a pre-trained language model trained on a large-scale corpus from the internet, with a Transformer encoder structure;

[0036] 2) Obtain the low-resource language pair dataset needed to build the neural machine translation system, and clean it;

[0037] 3) Obtain language analysis tools that can identify keywords containing a large amount of information in the text;

[0038] 4) Use language analysis tools to analyze the training data, mask the keywords, and fine-tune the pre-trained language model using the processed data, such as... Figure 1 As shown;

[0039] 5) Build a neural machine translation model, including using a fine-tuned pre-trained language model as the encoder of the neural machine translation model, and using a Transformer decoder with randomly initialized parameters as the decoder of the neural machine translation model.

[0040] 6) Use a three-stage training method to gradually train the encoder and decoder of the neural machine translation model;

[0041] 7) Deploy the trained neural machine translation model on a high-performance cloud server, and build the front-end and back-end to complete the construction of the entire machine translation system.

[0042] In step 1), a large-scale pre-trained language model with a Transformer encoder as the model structure is selected to be suitable for the neural machine translation model.

[0043] In step 2), the low-resource language pair dataset required by the user is obtained, and the dataset is cleaned using length ratio algorithm, HTML tag filtering, and special character filtering.

[0044] In step 3), language analysis tools capable of identifying text keywords are obtained, specifically including: stop word list and syntax tree parsing tools.

[0045] In step 4), the training data is first analyzed using language analysis tools to mask the keywords. Then, the pre-trained language model is fine-tuned using the processed data. The specific steps are as follows:

[0046] 401) For each sentence in the training corpus, use a stop word list to filter out words that are not stop words, and use the filtered words as the keyword set M, such as... Figure 2 The sentences in the text, after filtering out stop words, yield the results "like", "reading", and "afternoon".

[0047] 402) For each sentence in the training corpus, use a syntax tree parsing tool to analyze the dependency syntax tree of each text, and select the root node and the first-level child nodes of the syntax tree. The words corresponding to these nodes are used as the keyword set N, such as... Figure 2 The sentences in the text, filtered using a syntax tree parsing tool, yielded the keywords "I", "like", and "reading".

[0048] 403) For each sentence in the training corpus, select words that simultaneously belong to both keyword set M and keyword set N as the final keywords, such as... Figure 2 The sentences in the text were filtered out and selected as the final keywords: "like" and "reading".

[0049] 404) For each sentence in the training corpus, calculate the proportion a of the final filtered keywords to all words. If a is less than K, mask all keywords while randomly masking non-keywords to make the masking proportion reach K. If a is equal to K, mask all keywords. If a is greater than K, randomly select a portion of the keywords to make the masking proportion reach K. The value of K is in the range of [0.15, 0.25].

[0050] 405) Fine-tune the pre-trained language model using masked corpus.

[0051] In step 5), the overall architecture of the neural machine translation model is an encoder-decoder structure, using a fine-tuned pre-trained language model as the encoder of the neural machine translation model and a Transformer decoder as the decoder of the neural machine translation model.

[0052] In step 6), a three-stage training method is used to progressively train the encoder and decoder of the neural machine translation, such as... Figure 3 As shown, specifically:

[0053] 601) First, train the decoder part of the neural machine translation model with randomly initialized parameters separately until convergence;

[0054] 602) Then simultaneously train the encoder of the neural machine translation model that already contains a large amount of language knowledge and the decoder that can be adapted to the pre-trained language model;

[0055] 603) Finally, train the neural machine translation model decoder separately.

[0056] This invention addresses low-resource neural machine translation applications by designing a method to enhance neural machine translation models based on pre-trained model language knowledge. The following are a series of experimental results used to verify the effectiveness of the above design.

[0057] First, for the effectiveness verification experiment of the three-stage training method, this invention still uses a regular Transformer model to train the model on the iwslt14 small dataset with a learning rate of 0.00005. The specific results are shown in Table 1. It can be seen that after the third stage of training, the model's score on the test set is improved by 0.5 BLEU compared to the two-stage training, and by 1.96 BLEU compared to the baseline Transformer model without pre-training knowledge enhancement, which is a very significant improvement.

[0058] Table 1. Experimental Verification of the Effectiveness of the Three-Stage Training Method

[0059]

[0060] Next, the effectiveness of the masked keyword model fine-tuning method was verified through experiments. After extensive testing, this invention found that using a third keyword extraction method—the intersection of keywords extracted by two external knowledge tools—achieved a performance improvement exceeding the baseline by 0.22 BLEU. Specific experimental results are shown in Table 2. This invention uses keywords extracted through both masked syntax trees and stop words to fine-tune the pre-trained model. Its effectiveness lies in the fact that the proportion of keywords extracted using both rules is 18%, very close to the mask ratio of the original masked language model, making it suitable for model learning.

[0061] Table 2. Validation Experiment of Masked Keyword Model Fine-tuning Method

[0062]

[0063] Finally, Table 3 summarizes the most important experimental results in this chapter and provides the corresponding reference baselines and a comparison with another well-known method, BERT-nmt, which uses pre-trained knowledge to enhance neural machine translation systems. As shown in Table 3, the two-stage training method has exceeded the Transformer baseline by 1.46 BLEU. After using the three-stage training method, the score exceeds that of similar work BERT-nmt. The three-stage training method combined with the keyword masking language model proposed in this invention can exceed the baseline by 2.25 BLEU. This shows that after the improvement of this invention, the performance of BERT-Decoder, which already performs well in machine translation tasks of minority languages, has been further improved (0.79 BLEU).

[0064] Table 3. Experiments verifying the effectiveness of this invention.

[0065]

[0066] In summary, this invention employs a method of enhancing neural machine translation systems with pre-trained model knowledge as a technical approach. It investigates how to enable the decoder of a neural machine translation system to more fully and efficiently utilize the knowledge of the pre-trained model, exploring a way to maximize the potential of the pre-trained model in neural machine translation tasks for less commonly taught languages. Finally, experiments verify the effectiveness of this invention.

Claims

1. A method for constructing a low-resource neural machine translation system based on pre-trained knowledge enhancement, characterized in that... Includes the following steps: 1) Obtain a pre-trained language model trained on a large-scale corpus from the internet, with a Transformer encoder structure; 2) Obtain the low-resource language pair dataset needed to build the neural machine translation system, and clean it to obtain the training data; 3) Obtain language analysis tools that can identify keywords in text; 4) Use language analysis tools to analyze the training data, mask the keywords, and fine-tune the pre-trained language model using the processed data; 5) Build a neural machine translation model, including using a fine-tuned pre-trained language model as the encoder of the neural machine translation model, and using a Transformer decoder with randomly initialized parameters as the decoder of the neural machine translation model. 6) Use a three-stage training method to progressively train the encoder and decoder of the neural machine translation model; 7) Deploy the trained neural machine translation model on a high-performance cloud server and build the front-end and back-end processing programs to complete the construction of the entire machine translation system; In step 4), language analysis tools are used to analyze the training data, masking the keywords. Then, the pre-trained language model is fine-tuned using the processed data. The specific steps are as follows: 401) For each sentence in the training corpus, use the stop word list to filter out words that are not stop words, and use the filtered words as the keyword set M; 402) For each sentence in the training corpus, use a syntax tree parsing tool to analyze the dependency syntax tree of each text, and select the root node and the first-level child nodes of the syntax tree, and use the words corresponding to these nodes as the keyword set N; 403) For each sentence in the training corpus, select words that belong to both keyword set M and keyword set N as the final keywords; 404) For each sentence in the training corpus, calculate the proportion 'a' of the final filtered keywords to all words. If a is less than K, then mask all keywords while randomly masking non-keywords to make the masking proportion reach K. If a equals K, then mask all keywords. If a is greater than K, then randomly select a portion of the keywords to make the masking proportion reach K. 405) Fine-tune the pre-trained language model using masked corpus.

2. The method for constructing a low-resource neural machine translation system based on pre-trained knowledge enhancement as described in claim 1, characterized in that: In step 2), the low-resource language pair dataset required by the user is obtained, and the dataset is cleaned using length ratio algorithm, HTML tag filtering, and special character filtering.

3. The method for constructing a low-resource neural machine translation system based on pre-trained knowledge enhancement as described in claim 1, characterized in that: In step 3), obtain language analysis tools that can identify text keywords, specifically including: obtaining open-source stop word lists and syntax tree parsing tools from the internet, downloading and installing them.

4. The method for constructing a low-resource neural machine translation system based on pre-trained knowledge enhancement as described in claim 1, characterized in that: In step 6), the encoder and decoder of the neural machine translation are trained step by step using a three-stage training method, specifically as follows: 601) First, train the decoder part of the neural machine translation model with randomly initialized parameters separately until convergence; 602) Then simultaneously train the encoder of the neural machine translation model that already contains a large amount of language knowledge and the decoder that can adapt to the pre-trained language model; 603) Finally, train the neural machine translation model decoder separately.

Images