Machine translation method with high resource language priority for multi-language translation

Through the two-stage training framework and the selective language perception pool method, the performance degradation problem caused by negative gradient interference in multilingual neural machine translation is solved, and the translation performance improvement in high-resource language is achieved, proving that high-resource language priority is Machine translation methods can effectively alleviate negative interference and improve multilingual translation performance.

CN119089910APending Publication Date: 2024-12-06BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410417430.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-04-08
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Multilingual neural machine translation has negative gradient interference in shared parameters, resulting in the problem of degradation of translation performance in high-resource languages.

Method used

Using a two-stage training framework, high-resource specific language training and low-resource transfer learning, a cross-language pre-trained encoder model and selective language-aware pool generates language-specific representations, and conducts joint training on high-resource and low-resource bilingual corpus , ensure gradient propagation through weighted average.

Benefits of technology

It significantly improves the performance of multilingual translation, especially in high resource translation direction, performs better than bilingual and multilingual baselines, alleviates negative gradient interference, and the method is easy to apply to different backbone models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089910A_ABST
    Figure CN119089910A_ABST
Patent Text Reader

Abstract

The invention provides a method flow for multi-language translation, namely, a high-resource language-first machine translation method (HLT-MT), which is characterized by comprising two stages of training frameworks, namely high-resource specific language training and low-resource transfer learning. The high-resource specific language training mainly comprises the following steps of: initializing a multi-language model by using a cross-language pre-training encoder model (XLM-R) to obtain a target language sharing feature, and generating a language specific representation in a determined translation direction (the representation is finally used for generating a target sentence). The low-resource transfer learning mainly comprises the following steps that low-resource target sentences are generated by using underlying features generated by shared parameters, and a multi-language model is jointly trained on a high-resource bilingual corpus Dh and a low-resource bilingual corpus D1. Experiments show that the method can effectively relieve the gradient negative interference problem caused by different languages, and has good significance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of machine translation based on deep learning. Aiming at joint training of shared parameters for multiple languages, a method is proposed which can effectively alleviate the negative interference of gradients caused by different languages. Background Art

[0002] Multilingual Neural Machine Translation (MNMT) technology is trained in multiple language pairs. Because it shares knowledge between multiple languages, it can support multiple translation directions in different languages ​​at the same time using a single model by sharing parameters, thereby reducing training costs. Therefore, it has attracted great attention from academia and industry. However, due to the negative interference of gradients between different translation directions, multilingual machine translation training has the problem of performance degradation in multiple translation directions in shared parameters, especially on high-resource languages. Summary of the invention

[0003] 1. A method flow for multilingual translation, namely, a high-resource language-first machine translation method (HLT-MT), characterized by comprising a two-stage training framework, high-resource specific language training and low-resource transfer learning.

[0004] S1. The high-resource specific language training step is as follows: the multilingual model is initialized with a cross-language pre-trained encoder model (XLM-R), the shared features of the target language are obtained, and a language-specific representation is generated under a determined translation direction, which is ultimately used to generate the target sentence.

[0005] S2, the low-resource transfer learning step: using the underlying features generated by the shared parameters to generate low-resource target sentences, h and low-resource bilingual corpus D l Jointly train multi-language models;

[0006] 2. According to the multi-language translation method of claim 1, it is characterized in that in step S1, the high-resource specific language training step includes the following operations:

[0007] S101. Train the model using the selection-based language-aware pool (SLP) only in high-resource directions.

[0008] S102: Initialize the multilingual model using the cross-language pre-trained encoder model (XLM-R)

[0009] S103. Given a source sentence containing u words and a target sentence containing v words The target language L is obtained through the multi-head attention mechanism model (Transformer) kShared features at the top of the decoder The formula is as follows:

[0010] S104, after obtaining the high resource language L k A series of decoded representations Afterwards, the language-shared representation is mapped to a language-specific representation through a language-specific pool with a selective mechanism. Given a translation direction L i →L k (1≤i,k≤K∧i≠k) and selective language-specific pools Related modules It is possible to generate language-specific representations. (The g function is a mapping function that depends only on the target language) Use Functions share features between languages Shift to language-specific features The equation is as follows:

[0011] S105, in the above formula The function definition is as follows: Here, f is the layer normalization function and σ is the ReLU activation function. is the projection matrix is the lower projection matrix.

[0012] S106. For each source sequence, a special target language symbol is specified as a prefix to indicate the translation direction of the model, so that the decoder can correctly generate the target sentence using the shared decoder parameters. Therefore, the embedding of the target language symbol is used to select the language-specific module from the language-aware pool (SLP) based on the selection mechanism. The selection function is defined as follows:

[0013] S107. In the above formula, (T dimension), E[L k ] indicates the target language L k Embedding of symbols. Mapping target embedding to vector in, yes The ith element of and SLP consists of T sub-networks. The sub-network with the highest probability will be selected to generate language-specific features.

[0014] S108. In order to solve the non-differentiable problem of the language-aware pool (SLP) based on the selection mechanism, weighted averaging is used to ensure that the gradient can be propagated to all language-specific modules. The specific implementation is as follows:

[0015] S109, among which, It is calculated by target embedding and softmax function.

[0016] S1010, using the learning matrix W g Embedding the goal into E[L k ] is mapped to a probability vector

[0017] S1011, And E[L k ] indicates L k The language embedding of e is the embedding size. is a vector The tth element of

[0018] S1012. In actual training, the two objective functions in S104 and S108 are used alternately with equal probability to learn the mapping function and generate language-specific features.

[0019] S1013, this representation will eventually be used to generate the target sentence Among them, W o ∈d e ×V is the output matrix, and V is the vocabulary size.

[0020] 3. According to the multilingual translation method of claim 1, it is characterized in that in step S2, the low-resource transfer learning step includes the following operations:

[0021] S201, after the high resource module is fully trained, the shared features generated by the shared parameters Θ is fed to the general layer θ u , thereby generating shared features for low-resource languages;

[0022] S202. All low-resource languages ​​share the same common layer, sharing features Projection to the final representation

[0023] S203, through Generate the target sentence and output a matrix W similar to S1013 o ;

[0024] S204, in the high-resource bilingual corpus D h and low-resource bilingual corpus D l The joint dataset D h ∪D l We jointly tune on a selection-based language-aware pool (SLP) for high-resource languages ​​and a general layer for low-resource languages: where θ is a shared parameter for all languages. The language-aware pool (SLP) based on the selection mechanism contains a list of language-specific layers for a set of high-resource languages ​​(HRLs), θ u It is a general layer for low-resource languages ​​(LRLs). BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a two-stage framework for the HLT-MT method;

[0026] Figure 2 Training steps for high-resource specific languages;

[0027] Figure 3 Transfer learning training steps for low-resource languages; DETAILED DESCRIPTION

[0028] This method is divided into two stages: high-resource specific language training and low-resource transfer learning.

[0029] S1. The high-resource specific language training step is as follows: the multilingual model is initialized with a cross-language pre-trained encoder model (XLM-R), the shared features of the target language are obtained, and the language-specific representation under the determined translation direction is generated, which will eventually be used to generate sentences in the target language.

[0030] S101. Initialize the multilingual model using the cross-language pre-trained encoder model (XLM-R), given a source sentence containing u words and a target sentence containing v words The target language L is obtained through the multi-head attention mechanism model (Transformer) k Shared features at the top of the decoder The formula is as follows:

[0031] S102, after obtaining the high resource language L kA series of decoded representations Afterwards, the language-shared representation is mapped to a language-specific representation through a language-specific pool with a selection mechanism. Given a translation direction L i →L k (1≤i,k≤K∧i≠k) and selective language-specific pools Related modules It is possible to generate language-specific representations. (The g function is a mapping function that depends only on the target language) Use Functions share features between languages Shift to language-specific features The equation is as follows:

[0032] S103. For each source sequence, a special target language symbol is assigned as a prefix to indicate the translation direction of the model. The embedding of the target language symbol is used to select a language-specific module from a language-aware pool (SLP) based on a selection mechanism. The selection function is defined as follows:

[0033] S104: Use weighted averaging to ensure that the gradient can be propagated to all language-specific modules. The specific implementation is as follows:

[0034] S105, using the learning matrix W g Embedding the goal into E[L k ] is mapped to a probability vector

[0035] S106, alternately use the two objective functions in S102 and S104 with equal probability to learn the mapping function and generate language-specific representation This representation is finally used to generate the target sentence

[0036] S2. After the high-resource module is fully trained, low-resource transfer learning is performed on the translation model. The specific steps are as follows.

[0037] S201, shared features generated by shared parameters Θ is fed to the general layer θ u , thereby generating shared features for low-resource languages;

[0038] S202, all low-resource languages ​​share the same common layer, and the shared features Projection to the final representation

[0039] S203, use Generate the target sentence and output the matrix W o .

[0040] S204, in the high-resource bilingual corpus D h and low-resource bilingual corpus D l The joint dataset D h ∪D l We jointly tune on a selection-based language-aware pool (SLP) for high-resource languages ​​and a general layer for low-resource languages:

[0041] To evaluate the proposed method, experiments are conducted on the WMT-10 and OPUS-100 datasets.

[0042] For WMT-10, the experiments use parallel datasets of different languages ​​in the WMT dataset to evaluate the model, and parallel data between English and 10 other languages, including French (Fr), Czech (Cs), German (De), Finnish (Fi), Latvian (Lv), Estonian Gujarati (Gu).

[0043] For OPUS-100, the experiment uses the OPUS-100 corpus for large-scale multilingual machine translation. OPUS-100 is an English-centric multilingual corpus covering 100 languages, which is randomly sampled from the OPUS collection. After removing 5 languages ​​with missing test sets, 94 English-centric test sets were obtained.

[0044] The method described in this patent is compared with bilingual and multilingual methods. For a fairer comparison, XLM-R and LS-MNMT are initialized by XLM-R. BiNMT is a bilingual Transformer model. MNMT is trained jointly in all directions, where the target language token is a prefix of the input sentence. mBART is an encoder-decoder pre-trained model and then fine-tuned on all corpora. XLMR is initialized by the pre-trained model XLM-R. LS-MNMT integrates language-specific layers for all languages ​​at the end of the decoder.

[0045] Transformer is used as the backbone model for all experiments. Adam (β1=0.9, β2=0.98) is used to train the multilingual models. The learning rate is set to 5e-4 and the warm-up steps are 4000. The models are trained using label smoothed cross entropy with a smoothing ratio of 0.1. The batch size is 4096 words on 64 Tesla V100 GPUs. For WMT-10, the multilingual model is first trained on 6 languages ​​and then fine-tuned on all languages. For OPUS100, the model is trained on more than 10K languages. The evaluation metric is case-sensitive de-tokenized sacreBLEU.

[0046] The method described in this patent significantly improves the multilingual baseline in 10 translation directions. In the past, in resource-rich scenarios, the performance of multilingual machine translation was lower than that of bilingual translation models. It is worth noting that the multilingual machine translation baseline XLM-R initialized by a cross-language pre-trained model is already very competitive. Interestingly, the method performs better than the bilingual baseline in high-resource translation directions, such as English to German (+1.5 BLEU points). The method consistently outperforms the multilingual baseline on all language pairs, confirming that using a high-resource language-first machine translation method (HLT-MT) to mitigate negative interference can help improve performance.

[0047] In addition, it can be observed from the experimental results that the high-resource language-first machine translation method (HLT-MT) achieved better results in 94 translation directions. This improvement is due to the selective language-specific pool in high-resource training, which avoids the competition between high-resource and low-resource training directions. Another advantage of this method is that it is lightweight and convenient and can be applied to different backbone models because the parameters of the selective language-specific pool at the top of the decoder are small compared to all the parameters.

Claims

1. A method flow for multilingual translation A high-resource language priority machine translation method (HLT-MT), characterized in that It includes a two-stage training framework: high-resource language-specific training and low-resource transfer learning. S1. The high-resource specific language training step is as follows: the multilingual model is initialized with a cross-language pre-trained encoder model (XLM-R), the shared features of the target language are obtained, and a language-specific representation is generated under a determined translation direction, which is ultimately used to generate the target sentence. S2, the low-resource transfer learning step: using the underlying features generated by the shared parameters to generate low-resource target sentences, h and low-resource bilingual corpus D l Jointly train multi-lingual models on 2. The method of multilingual translation according to claim 1, characterized in that: In step S1, the high-resource specific language training step includes the following operations: S101. Train the model using the selection-based language-aware pool (SLP) only in high-resource directions. S102: Initialize the multilingual model using the cross-language pre-trained encoder model (XLM-R) S103. Given a source sentence containing u words and a target sentence containing v words The target language L is obtained through the multi-head attention mechanism model (Transformer) k Shared features at the top of the decoder The formula is as follows: S104, after obtaining the high resource language L k A series of decoded representations Afterwards, the language-shared representation is transformed into language-specific representation through a language-specific pool with a selective mechanism. Given a translation direction L i →L k (1≤i,k≤K∧i≠k) and selective language-specific pools Related modules It is possible to generate language-specific representations. (The g function is a mapping function that depends only on the target language) Use Functions share features between languages Shift to language-specific features The equation is as follows: S105, in the above formula The function definition is as follows: Here, f is the layer normalization function and σ is the ReLU activation function. is the projection matrix is the lower projection matrix. S106. For each source sequence, a special target language symbol is specified as a prefix to indicate the translation direction of the model, so that the decoder can correctly generate the target sentence using the shared decoder parameters. Therefore, the embedding of the target language symbol is used to select the language-specific module from the language-aware pool (SLP) based on the selection mechanism. The selection function is defined as follows: S107. In the above formula, (T dimension), E[L k ] indicates the target language L k Embedding of symbols. Mapping target embedding to vector in, yes The ith element of and SLP consists of T sub-networks. The sub-network with the highest probability will be selected to generate language-specific features. S108. In order to solve the non-differentiable problem of the language-aware pool (SLP) based on the selection mechanism, weighted averaging is used to ensure that the gradient can be propagated to all language-specific modules. The specific implementation is as follows: S109, among which, It is calculated by target embedding and softmax function. S1010, using the learning matrix W g Embedding the goal into E[L k ] is mapped to a probability vector S1011、 And E[L k ] indicates L k The language embedding of e is the embedding size. is a vector The tth element of S1012. In actual training, the two objective functions in S104 and S108 are used alternately with equal probability to learn the mapping function and generate language-specific representations. S1013, this representation will eventually be used to generate the target sentence Among them, W o ∈d e ×V is the output matrix, and V is the vocabulary size.

3. The method of multilingual translation according to claim 1, characterized in that: In step S2, the low-resource transfer learning step includes the following operations: S201, after the high resource module is fully trained, the shared features generated by the shared parameters Θ is fed to the general layer θ u , thereby generating shared features for low-resource languages; S202. All low-resource languages ​​share the same common layer and share the shared features. Projection to the final representation S203, through Generate the target sentence and output a matrix W similar to S1013 o ; S204, in the high-resource bilingual corpus D h and low-resource bilingual corpus D l The joint dataset D h ∪D l We jointly tune on a selection-based language-aware pool (SLP) for high-resource languages ​​and a general layer for low-resource languages: where θ is a shared parameter for all languages. The language-aware pool (SLP) based on the selection mechanism contains a list of language-specific layers for a set of high-resource languages ​​(HRLs), θ u It is a general layer for low-resource languages ​​(LRLs).