Long-distance language neural machine translation method based on cross-language unified word representation
The method addresses vocabulary mismatches in long-distance languages by using cross-language unified word representation to align and constrain training, significantly improving translation performance in low-resource scenarios.
Patent Information
- Application Number
- CN202411109377.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-12-06
AI Technical Summary
Neural machine translation methods face performance degradation due to vocabulary mismatches between long-distance languages with different writing systems and grammatical differences, limiting the effectiveness of transfer learning in low-resource scenarios.
A long-distance language neural machine translation method based on cross-language unified word representation, utilizing professional bilingual dictionaries to build a unified vocabulary, forming bilingual parallel sentence pairs, and constraining model training with original target language features to enhance vocabulary alignment and performance.
Improves translation quality by aligning distant language representations, enhancing transfer learning effectiveness, and achieving significant improvements in BLEU scores compared to baseline models.
Smart Images

Figure CN119089913A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a long-distance language neural machine translation method based on cross-language unified word representation, and belongs to the field of natural language processing. Background Art
[0002] Compared with statistical machine translation methods, neural network-based machine translation methods have been proven to be more effective and have become the mainstream method in the current machine translation field. However, NMT is a data-driven method, and the quality of its translation results depends on the amount of large-scale data sets, which leads to a large difference in its performance on low-resource language pairs and high-resource language pairs. Transfer learning transfers knowledge from high-resource language models to low-resource models, which is an effective way to improve low-resource neural machine translation.
[0003] In low-resource scenarios, long-distance language pairs are a special case. For example, Chinese and Urdu, Vietnamese, and Lao have very different written symbols. Even Urdu is written from right to left. For two languages that use different writing systems and have obvious differences in the grammar and semantics of words, their semantic representations of words are usually quite different, resulting in low word list overlap. The word list mismatch problem may limit the effect of transfer learning.
[0004] Recently, Aji et al. pointed out that in order to effectively transfer the knowledge of the embedding layer, the parent model and the child model need to use the same or partially overlapping vocabulary. To this end, Lakew et al. proposed a method to merge the vocabulary by aligning the same sub-vocabularies or randomly assigning unseen sub-vocabularies. However, the above methods perform well when migrating between languages with the same writing system, but the results are not satisfactory between different writing systems or distant languages with different civilizations. This brings additional challenges to the application of transfer learning in low-resource fields. Amrhein et al. proposed that romanization tools may cause information loss, but it can significantly improve the transfer effect between distant languages. However, even if an additional information restoration step is introduced, the information loss problem caused by the romanization method is still difficult to completely alleviate, thereby affecting the final performance of transfer learning. In response to the above problems, the present invention proposes a long-distance language neural machine translation method based on cross-language unified word representation. Summary of the invention
[0005] To solve the above problems, the present invention proposes a long-distance language neural machine translation method based on cross-language unified word representation to solve the performance degradation problem caused by the mismatch of vocabulary between long-distance languages when the low-resource neural machine translation model uses the transfer learning method. The present invention has achieved good experimental results in the transfer learning experiment of migrating the English-Vietnamese machine translation model to the Chinese-Vietnamese machine translation model.
[0006] The technical solution of the present invention is: a long-distance language neural machine translation method based on cross-language unified word representation, and the specific steps of the method are as follows:
[0007] Step 1, build a cross-language unified word representation vocabulary: use professional bilingual dictionaries to build a cross-language unified word representation vocabulary to unify the source language and the target language, and build a cross-language unified word representation vocabulary including the source language, the target language and English;
[0008] Step 2, build unified word representation data: replace the corresponding words in the target language according to the cross-language unified word representation vocabulary to build unified word representation data, and form bilingual parallel sentence pairs with the source language data and the unified word representation data as the training data of the sub-model. At the same time, save the data before the unified word representation operation to generate feature constraint data;
[0009] Step 3. Build the parent model of the transfer learning method: Based on the Transformer model, use the English-target language bilingual parallel data as training data, train the parent model, and save the best model parameters to initialize the child model and generate feature constraint data;
[0010] Step 4. Build a long-distance language neural machine translation model based on cross-language unified word representation and use it for translation: Use the constructed unified word representation data as the input of the neural translation model. During the training of the neural translation model, the sentence features of the original data of the target language before the unified word representation are used to constrain the training of the model to reduce the unified word representation to ensure that the model can learn the correct target language features. Finally, an additional restoration module is introduced to restore the unified word representation data output by the model to the target language to obtain the final translation result.
[0011] As a further solution of the present invention, the specific steps of Step 1 are as follows:
[0012] Step 1.1. First, use a regular expression-based tagger to preprocess the electronic professional dictionary data and extract the bilingual corresponding entries in the electronic professional dictionary from the electronic professional dictionary data. Secondly, screen and filter the extracted bilingual corresponding entries and delete empty or missing entries. Finally, according to the corresponding relationship of the entries, the professional electronic dictionary is divided into a one-to-one word table and a one-to-many word table;
[0013] Step 1.2, directly retain the one-to-one word table obtained in Step 1.1 as part of the unified word representation word table;
[0014] Step 1.3: For the one-to-many vocabulary obtained in Step 1.1, use the existing high-performance machine translation model to translate the source language of each entry into English, form a correspondence between the source language and English, and use it as part of the unified word representation vocabulary;
[0015] Step 1.4: Merge the vocabularies obtained in Step 1.2 and Step 1.3 to form the final unified word representation vocabulary.
[0016] As a further solution of the present invention, Step 2 includes the following:
[0017] Step 2.1, using the word correspondence in the unified word representation vocabulary obtained in Step 1, replace the vocabulary of each sentence in the target language data to obtain sentences in the form of code switching (code switching, csw), and use them as unified word representation data;
[0018] Step 2.2, the unified word representation data obtained in Step 2.1 and the source language data are combined to form a bilingual parallel sentence pair, wherein the source language data is still used as the source language in the bilingual parallel sentence pair, and the unified word representation data is used as the target language in the bilingual parallel sentence pair;
[0019] Step 2.3, the target language data aligned with the source language in the bilingual parallel sentence pairs obtained in Step 2.2, that is, the original target language before the unified word representation in Step 2.1, is stored in the order aligned with the source language to generate feature constraint data.
[0020] As a further solution of the present invention, Step 3 includes the following:
[0021] Step 3.1. Train the parent model: Based on the Transformer model, use English-target language language pair data to train the parent model, and save the best trained model as a ".pt" format file as the parent model.
[0022] As a further solution of the present invention, the specific steps of Step 4 are as follows:
[0023] Step 4.1, construct feature constraint data: for each sentence pair (x c ,y c )∈(X c ,Y c ), use Step 3.1 to get the parent model to convert the target language sentence y of the child model c Back translated to And get the pseudo parent data
[0024] Step 4.2, training sub-model: Based on the Transformer model, use the bilingual parallel sentence pairs obtained in Step 2.2 as the model input to train the model. At the same time, use Step 4.1 to obtain the sentence representation of the corresponding sentence in the feature constraint data, and calculate the mean square error loss together with the output features of the sub-model inference results, and use the sum of the cross entropy loss obtained from the model training and the calculated mean square error loss as the final error of the model training, so as to constrain the model to learn the correct target language features;
[0025] Specifically, for each sentence pair of pseudo parent data The parent model is the token of the original sentence before each csw replacement Generate feature labels The sub-model also generates feature labels for the corresponding csw format data tags Afterwards, this paper uses mean-square error (MSE) loss to calculate the square of the Euclidean distance between the two feature labels output by the parent model and the child model:
[0026]
[0027] Among them, θ^p and θ^c represent the parameters of the parent model and the child model respectively.
[0028] The loss of the final sub-model is a linear combination of the cross entropy loss (CE) and the mean squared error loss with a hyperparameter α:
[0029] L=L CE +αL MSE (2)
[0030] Step 4.3, output result restoration: In order to obtain the expected final result of the machine translation task, an additional machine translation model is used as the result restoration model. The inference result output by the sub-model is used as the source language, and the original target language saved in Step 2.3 is used as the target language to form bilingual parallel sentence pairs. Based on the Transformer model, the above-mentioned bilingual parallel sentence pairs are input into the model as training data for training. The best model obtained by training is the final result restoration model. The unified word representation data output by the sub-model reasoning is used as the input of the result restoration model to obtain the output of the result restoration model, which is the final machine translation result in the target language.
[0031] The beneficial effects of the present invention are:
[0032] 1. The present invention uses a professional bilingual dictionary to construct a unified word representation vocabulary to enable texts between distant languages to have better consistency in representation;
[0033] 2. The present invention uses the data before unified word representation to constrain the training process of the sub-model, so that the sub-model can learn the correct target language features from the unified word representation data in the csw format.
[0034] 3. The present invention is based on a long-distance language neural machine translation method with cross-language unified word representation. It uses a constructed unified word representation vocabulary to improve the degree of vocabulary alignment between long-distance languages, enhance the migration effect from the parent model to the child model of the transfer learning method, and thus improve the performance of the low-resource neural machine translation model. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is an overall flow chart of the long-distance language neural machine translation method based on cross-language unified word representation in the present invention;
[0036] Figure 2 This is a model architecture diagram of the long-distance language neural machine translation method based on cross-language unified word representation in the present invention. DETAILED DESCRIPTION
[0037] Example 1: Figure 1-Figure 2 As shown, a long-distance language neural machine translation method based on cross-language unified word representation, the specific steps of the method are as follows:
[0038] Step 1, construct a cross-language unified word representation vocabulary: Use a professional bilingual dictionary to construct a cross-language unified word representation vocabulary to unify the source language and the target language, and construct a cross-language unified word representation vocabulary including the source language, the target language and English. The specific steps of Step 1 are as follows:
[0039] Step 1.1. First, use a regular expression-based tagger to preprocess the electronic professional dictionary data and extract the bilingual corresponding entries in the electronic professional dictionary from the electronic professional dictionary data. Secondly, screen and filter the extracted bilingual corresponding entries and delete empty or missing entries. Finally, according to the corresponding relationship of the entries, the professional electronic dictionary is divided into a one-to-one word table and a one-to-many word table;
[0040] Step 1.2, directly retain the one-to-one word table obtained in Step 1.1 as part of the unified word representation word table;
[0041] Step 1.3: For the one-to-many vocabulary obtained in Step 1.1, use the existing high-performance machine translation model to translate the source language of each entry into English, form a correspondence between the source language and English, and use it as part of the unified word representation vocabulary;
[0042] Step 1.4: Merge the vocabularies obtained in Step 1.2 and Step 1.3 to form the final unified word representation vocabulary.
[0043] Step 2, build unified word representation data: replace the corresponding words in the target language according to the cross-language unified word representation vocabulary to build unified word representation data, and form bilingual parallel sentence pairs with the source language data and the unified word representation data as the training data of the sub-model. At the same time, save the data before the unified word representation operation to generate feature constraint data;
[0044] As a further solution of the present invention, Step 2 includes the following:
[0045] Step 2.1, using the word correspondence in the unified word representation vocabulary obtained in Step 1, replace the vocabulary of each sentence in the target language data to obtain sentences in the form of code switching (code switching, csw), and use them as unified word representation data;
[0046] Table 1
[0047]
[0048] Step 2.2, the unified word representation data obtained in Step 2.1 and the source language data are combined to form a bilingual parallel sentence pair, wherein the source language data is still used as the source language in the bilingual parallel sentence pair, and the unified word representation data is used as the target language in the bilingual parallel sentence pair;
[0049] Step 2.3, the target language data aligned with the source language in the bilingual parallel sentence pairs obtained in Step 2.2, that is, the original target language before the unified word representation in Step 2.1, is stored in the order aligned with the source language to generate feature constraint data.
[0050] Step 3. Build the parent model of the transfer learning method: Based on the Transformer model, use the English-target language bilingual parallel data as training data, train the parent model, and save the best model parameters to initialize the child model and generate feature constraint data;
[0051] As a further solution of the present invention, Step 3 includes the following:
[0052] Step 3.1. Train the parent model: Based on the Transformer model, use English-target language language pair data to train the parent model, and save the best trained model as a ".pt" format file as the parent model.
[0053] Step 4. Build a long-distance language neural machine translation model based on cross-language unified word representation and use it for translation: Use the constructed unified word representation data as the input of the neural translation model. During the training of the neural translation model, the sentence features of the original data of the target language before the unified word representation are used to constrain the training of the model to reduce the unified word representation to ensure that the model can learn the correct target language features. Finally, an additional restoration module is introduced to restore the unified word representation data output by the model to the target language to obtain the final translation result.
[0054] As a further solution of the present invention, the specific steps of Step 4 are as follows:
[0055] Step 4.1, construct feature constraint data: for each sentence pair (x c ,y c )∈(X c ,Y c ), use Step 3.1 to get the parent model to convert the target language sentence y of the child model c Back translated to And get the pseudo parent data
[0056] Step 4.2, training sub-model: Based on the Transformer model, use the bilingual parallel sentence pairs obtained in Step 2.2 as the model input to train the model. At the same time, use Step 4.1 to obtain the sentence representation of the corresponding sentence in the feature constraint data, and calculate the mean square error loss together with the output features of the sub-model inference results, and use the sum of the cross entropy loss obtained from the model training and the calculated mean square error loss as the final error of the model training, so as to constrain the model to learn the correct target language features;
[0057] Specifically, for each sentence pair of pseudo parent data The parent model is the token of the original sentence before each csw replacement Generate feature labels The sub-model also generates feature labels for the corresponding csw format data tags Afterwards, the present invention uses mean-square error (MSE) loss to calculate the square of the Euclidean distance between the two feature labels output by the parent model and the child model:
[0058]
[0059] Among them, θ^p and θ^c represent the parameters of the parent model and the child model respectively.
[0060] The loss of the final sub-model is a linear combination of the cross entropy loss (CE) and the mean squared error loss with a hyperparameter α:
[0061] L=L CE +αL MSE (2)
[0062] Step 4.3, output result restoration: In order to obtain the expected final result of the machine translation task, an additional machine translation model is used as the result restoration model. The inference result output by the sub-model is used as the source language, and the original target language saved in Step 2.3 is used as the target language to form bilingual parallel sentence pairs. Based on the Transformer model, the above-mentioned bilingual parallel sentence pairs are input into the model as training data for training. The best model obtained by training is the final result restoration model. The unified word representation data output by the sub-model reasoning is used as the input of the result restoration model to obtain the output of the result restoration model, which is the final machine translation result in the target language.
[0063] In order to illustrate the effect of the present invention, the present invention selects English-Vietnamese (hereinafter referred to as "English-Vietnamese") as the parent language pair and Chinese-Vietnamese (hereinafter referred to as "Chinese-Vietnamese") as the child language pair, and verifies the model proposed by the present invention under the parent-child framework of transfer learning.
[0064] The present invention uses the Vietnamese-Chinese dictionary of the DeepBlue electronic dictionary, and after screening the valid data in the electronic data, a total of 88,960 valid entries are extracted, including 58,403 one-to-one entries and 30,557 one-to-many entries.
[0065] The present invention uses the English-Vietnamese dataset of about 50 million sentence pairs and the Chinese-Vietnamese dataset of about 8 million sentence pairs in the CCMatrix dataset. After cleaning, filtering and screening, 8 million sentence pairs of high-quality English-Vietnamese corpus were finally extracted, including 3,000 test corpus and 3,000 verification corpus; the selection of Chinese-Vietnamese corpus also extracted 1 million sentence pairs according to the same standard, of which 3,000 sentences were test corpus and 3,000 sentences were verification corpus.
[0066] The present invention uses the Fairseq toolkit to implement the model, and selects the Transformer model with 6 layers of encoders and decoders as the basic architecture of the machine translation experiment. Specific parameter settings: the word embedding vector dimension and the hidden layer dimension are both 512, the learning rate is set to 9e-5, and α is set to 0.01; Adam is used as the optimizer, and the optimizer is set to β_1=0.9, β_2=0.98. To prevent the model from overfitting, the label smoothing value is 0.1; the decoding process uses a constrained search algorithm, and the search width is set to 5; BLEU is used as the evaluation indicator. The Dropout value of the parent model is 0.1, and 150 epochs are trained; the Dropout value of all child models is 0.3, and all are trained for 150 epochs. The entire training process is performed on an NVIDIA RTX 3090.
[0067] In order to verify the effectiveness of the proposed cross-language unified word representation model, the present invention conducted the following five groups of experiments. First, without using the transfer learning strategy, the NMT model was trained on the basic Transformer model using the sub-model language pair bilingual data. At the same time, the NMT model based on the transfer learning strategy was trained on the KNN-TL model using the parent model and sub-model language pair bilingual data, and the basic Transformer model and the KNN-TL model were used as baseline models for comparison. Next, based on the model proposed in the present invention, the transfer learning model was trained using the parent model and sub-model language pair bilingual data, and during the training process, sub-model training data with different word segmentation strategies were used respectively. The specific experimental results are shown in Table 2.
[0068] According to the experimental results in Table 2, the KNN-TL model achieved a BLEU value improvement of 7.5 in the Chinese-Vietnamese translation direction compared to the basic Transformer model, which once again proves the effectiveness of the transfer learning parent-child framework. Under the transfer learning parent-child framework, the model proposed in the present invention uses an appropriate word segmentation method. Both methods achieve significant improvements over the KNN-TL model trained based on the transfer learning parent-child framework, proving the effectiveness of incorporating a cross-language unified word representation model. The model proposed in the present invention achieved BLEU value improvements of 1.35 and 2.53 respectively compared to the baseline KNN-TL model in the cases of "csw form data uses bert word segmentation, target language data uses bpe word segmentation" and "csw form data and target language data both use bert word segmentation".
[0069] Table 2
[0070]
[0071] In addition, in the experimental results of Table 2, the BLEU value of the model proposed by the present invention in the case of "both the csw form data and the target language data use bert participle" is 1.18 BLEU values higher than the BLEU value in the case of "the csw form data uses bert participle, and the target language data uses bpe participle". In response to this phenomenon, the present invention believes that the reason for this phenomenon is that the encoding results of the bpe participle strategy and the bert participle strategy are different, which is equivalent to two artificially created languages with different writing systems. If the original sentence of the target language after the bpe participle is used to constrain the sentence in the csw form after the bert participle, this difference in writing system will increase the burden of model learning, so that the model needs to learn more knowledge to obtain the relationship between the two representations. In contrast, the original sentence of the target language after the bert participle is used to constrain the sentence in the csw form after the bert participle, and the representation forms of the two sentences are closer, and the model can more easily learn the characteristic information of the original sentence of the target language.
[0072] The present invention uses a replacement operation when integrating the unified word representation vocabulary into the target language end and constructing the unified word representation data. Next, the present invention will explore the impact of using different replacement strategies on the one-to-one word table and the one-to-many word table on the model performance. The one-to-one word table is the reserved Vietnamese-Chinese word correspondence, and the one-to-many word table is the Vietnamese-English secondary correspondence. By controlling the number of replacement operations and the number of vocabularies used in the replacement operation, the present invention tested different replacement strategies. The experimental results are shown in Table 3.
[0073] Table 3
[0074]
[0075] The results in Table 3 show that when the replacement strategy is "not distinguishing one-to-many or one-to-one vocabulary", the model proposed in the present invention achieves the best model effect. When the second-phase replacement strategy is the other three strategies in the table, the model performance does not exceed the model performance when "not distinguishing one-to-many or one-to-one vocabulary". In addition, when the replacement strategy is "only in one-to-one vocabulary", the model score is close to the highest score, and the scores on the strategies of "both in two vocabularies" and "only in one-to-many vocabulary" are improved by more than 1 BLEU value. This shows that when constructing unified word representation data, the diversity of the data can provide more information for the model and is more robust during testing.
[0076] The experimental model of the present invention is used as a Chinese-Vietnamese NMT model. When the model only accepts the knowledge of the English-Vietnamese vocabulary, the model's score is relatively low due to the lack of relevant knowledge of Chinese and Vietnamese. Secondly, under the "two vocabularies at the same time" strategy, that is, all training data of the model must be data that includes Chinese, English and Vietnamese at the same time, the model lacks relevant knowledge for sentences composed only of Chinese and Vietnamese vocabulary, and the performance is not ideal. Finally, when the model only accepts sentences composed of Chinese and Vietnamese vocabulary, although this sentence form meets the translation direction of the model and scores higher than the aforementioned two forms of data, the forced integration of two distant languages into the same sentence leads to increased information loss, which in turn leads to poor robustness of the model, and slightly worse performance than the "no distinction between vocabulary" strategy.
[0077] It can be seen that the present invention proposes an NMT model based on cross-language unified word representation to alleviate the performance degradation problem caused by the mismatch of the vocabulary between distant languages when the low-resource NMT model uses the transfer learning method. First, the present invention constructs a unified word representation vocabulary by using a professional bilingual dictionary, and uses the vocabulary to generate data for unified word representation. Secondly, the present invention uses the unified word representation data and the source language sentence data to form bilingual parallel data, making it the training data of the sub-model, and inputs the model for training. Finally, the present invention uses the characteristics of the original target language sentence to constrain the training process. In addition, the present invention performs an additional restoration operation on the translation result of the model to obtain the final result expected by the machine translation task. Experimental results show that the model proposed by the present invention has achieved remarkable results in alleviating the problem of limited performance of transfer learning under distant languages, and has improved the BLEU value by 2.53 compared with the baseline model.
[0078] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A long-distance language neural machine translation method based on cross-language unified word representation, characterized by: The specific steps of the method are as follows: Step 1, build a cross-language unified word representation vocabulary: use professional bilingual dictionaries to build a cross-language unified word representation vocabulary to unify the source language and the target language, and build a cross-language unified word representation vocabulary including the source language, the target language and English; Step 2, construct unified word representation data: replace the corresponding words in the target language according to the cross-language unified word representation vocabulary to construct unified word representation data, and form bilingual parallel sentence pairs with the source language data and the unified word representation data as the training data of the sub-model; at the same time, save the data before the unified word representation operation to generate feature constraint data; Step 3. Build the parent model of the transfer learning method: Based on the Transformer model, use the English-target language bilingual parallel data as training data, train the parent model, and save the best model parameters to initialize the child model and generate feature constraint data; Step 4. Build a long-distance language neural machine translation model based on cross-language unified word representation and use it for translation: Use the constructed unified word representation data as the input of the neural translation model. During the training of the neural translation model, use the sentence features of the original data of the target language before unified word representation to constrain the training of the model to reduce the unified word representation to ensure that the model can learn the correct target language features. Finally, introduce an additional restoration module to restore the unified word representation data output by the model to the target language to obtain the final translation result.
2. The long-distance language neural machine translation method based on cross-language unified word representation according to claim 1 is characterized by: The specific steps of Step 1 are as follows: Step 1.
1. First, use a regular expression-based tagger to preprocess the electronic professional dictionary data and extract the bilingual corresponding entries in the electronic professional dictionary from the electronic professional dictionary data; secondly, screen and filter the extracted bilingual corresponding entries and delete empty or missing entries. Finally, according to the corresponding relationship of the entries, the professional electronic dictionary is divided into a one-to-one word table and a one-to-many word table; Step 1.2, directly retain the one-to-one word table obtained in Step 1.1 as part of the unified word representation word table; Step 1.3: For the one-to-many vocabulary obtained in Step 1.1, use the existing high-performance machine translation model to translate the source language of each entry into English, form a correspondence between the source language and English, and use it as part of the unified word representation vocabulary; Step 1.4: Merge the vocabularies obtained in Step 1.2 and Step 1.3 to form the final unified word representation vocabulary.
3. The long-distance language neural machine translation method based on cross-language unified word representation according to claim 1 is characterized in that: The specific steps of Step 2 are as follows: Step 2.1, using the word correspondence in the unified word representation vocabulary obtained in Step 1, replace the vocabulary of each sentence in the target language data to obtain the sentence in code-converted form, and use it as the unified word representation data; Step 2.2, the unified word representation data obtained in Step 2.1 and the source language data are combined to form a bilingual parallel sentence pair, wherein the source language data is still used as the source language in the bilingual parallel sentence pair, and the unified word representation data is used as the target language in the bilingual parallel sentence pair; Step 2.3, the target language data aligned with the source language in the bilingual parallel sentence pairs obtained in Step 2.2, that is, the original target language before the unified word representation in Step 2.1, is stored in the order aligned with the source language to generate feature constraint data.
4. The long-distance language neural machine translation method based on cross-language unified word representation according to claim 1, characterized in that: The Step 3 includes: Step 3.
1. Train the parent model: Based on the Transformer model, use English-target language language pair data to train the parent model, and save the best trained model as a ".pt" format file as the parent model.
5. The long-distance language neural machine translation method based on cross-language unified word representation according to claim 1, characterized in that: The specific steps of Step 4 are as follows: Step 4.1, construct feature constraint data: for each sentence pair (x c ,y c )∈(X c ,Y c ), use the obtained parent model to transform the target language sentence y of the child model c Back translated to And get the pseudo parent data Step 4.2, training sub-model: Based on the Transformer model, use the obtained bilingual parallel sentence pairs as the model input to train the model; at the same time, use Step 4.1 to obtain the sentence representation of the corresponding sentence in the feature constraint data, and calculate the mean square error loss together with the output features of the sub-model inference results, and use the sum of the cross entropy loss obtained from the model training and the calculated mean square error loss as the final error of the model training, so as to constrain the model to learn the correct target language features; Specifically, for each sentence pair of pseudo parent data The parent model is the token of the original sentence before each csw replacement Generate feature labels The sub-model also generates feature labels for the corresponding csw format data tags Afterwards, the mean squared error loss is used to calculate the square of the Euclidean distance between the two feature labels output by the parent model and the child model: Among them, θ^p and θ^c represent the parameters of the parent model and the child model respectively; The loss of the final sub-model is a linear combination of the cross entropy loss and the mean squared error loss with a hyperparameter α: L=L CE +αL MSE (2) Step 4.3, output result restoration: In order to obtain the desired final result of the machine translation task, an additional machine translation model is used as the result restoration model; the inference result output by the sub-model is used as the source language, and the original target language saved in Step 2.3 is used as the target language to form bilingual parallel sentence pairs; based on the Transformer model, the above-mentioned bilingual parallel sentence pairs are input as training data into the model for training; The best model obtained through training is the final result restoration model; the unified word representation data output by the sub-model reasoning is used as the input of the result restoration model, and the output of the result restoration model is the final machine translation result in the target language.
