A multilingual neural machine translation model training method based on meta-learning and language pair subnetwork

Abstract

A multilingual neural machine translation model training method based on meta-learning and language pair subnets belongs to the field of machine translation technology. The present invention solves the problem of poor translation performance of existing multilingual translation sharing models. The present invention optimizes the initial parameters of the multilingual translation model with the help of ML methods to obtain local optimal parameter values. ML can reduce parameter interference and ensure translation performance by sharing representations between learning tasks and adaptively adjusting each language pair. The mask matrix of each language pair task can be obtained by learning language-specific subnets. During model training, only the model parameters corresponding to the input language pair data are updated each time, so that the multilingual translation model can avoid parameter interference and improve training efficiency. In subsequent translation tasks, only the model parameters corresponding to element 1 in the mask matrix are required to participate in the prediction of translation results, which reduces the complexity of model calculation. The method of the present invention can be applied to the field of machine translation.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation, and in particular relates to a multilingual neural machine translation model training method based on meta-learning (ML) and language pair subnets. Background Art

[0002] Due to the wide variety of languages ​​used by humans, language barriers can easily arise for those unfamiliar with multiple languages. With the development of deep learning, traditional neural machine translation (NMT) methods typically train separate translation models for each language pair (source and target). Each model independently learns the mapping relationship between the source and target languages. While this approach can provide high-quality translation results, the computational and storage costs rise sharply as the number of language pairs increases. Especially when training multiple language pairs, redundant computation and storage make model scalability difficult. Multilingual neural machine translation (MNMT) processes multiple language pairs by sharing a model, reducing the computational burden of separate training for each language pair in traditional NMT methods. In multilingual translation tasks, multiple language pairs share a single translation model, but as the number of tasks increases, the model parameters become complex (they fail to consider the relationships between tasks and the specific circumstances within a task).

[0003] Therefore, the existing multilingual translation sharing model still has the following problems:

[0004] (1) The task differences between different language pairs lead to parameter interference in the multilingual translation sharing model, that is, the parameter optimization of some language pairs will affect the translation effect of other language pairs (especially the data imbalance between high-resource and low-resource languages).

[0005] (2) There are redundant parameters in the multilingual translation sharing model, which impairs the performance of single-language translation. Summary of the Invention

[0006] The purpose of the present invention is to solve the problem of poor translation performance of the existing multilingual translation sharing model and propose a multilingual neural machine translation model training method based on meta-learning and language pair subnets.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a multilingual neural machine translation model training method based on meta-learning and language pair subnet, the method specifically comprising the following steps:

[0008] Step 1: Obtain a multi-language dataset, preprocess it, and build a multi-language global shared dictionary based on the pre-processed multi-language pairs. Add a language category identifier to the pre-processed multi-language pairs, and then divide the pre-processed multi-language pairs into three parts: training set, validation set, and test set.

[0009] Step 2: Pre-train the multilingual neural machine translation model based on the meta-learning method, the multilingual global shared dictionary, and the training set, validation set, and test set to obtain a pre-trained multilingual neural machine translation model;

[0010] Step 3: Build a subnet for each language pair in the multilingual dataset. The structure of the subnet for each language pair is the same as that of the multilingual neural machine translation model. Each subnet is loaded with the parameters of the pre-trained multilingual neural machine translation model.

[0011] For any language pair in the multi-language dataset, fine-tune the parameters of the subnet after loading the parameters using the preprocessing results corresponding to the language pair, and generate the mask matrix of the language pair based on the fine-tuning results;

[0012] Step 4: Use the preprocessed results corresponding to each language pair to train the pre-trained multilingual neural machine translation model to obtain a trained multilingual neural machine translation model;

[0013] When the pre-trained model is trained using the pre-processed results corresponding to the i-th language pair, only the parameter weight corresponding to element 1 in the mask matrix corresponding to the i-th language pair is adjusted;

[0014] Step 5: After preprocessing the sentence to be translated, add a language category identifier to the preprocessing result, and then use the preprocessing result with the added language category identifier as the input of the trained multilingual neural machine translation model, and output the translation result through the trained multilingual neural machine translation model.

[0015] Furthermore, the multi-language pair dataset is downloaded from IWSLT14, and the multi-language pair dataset specifically includes parallel corpora consisting of English and French, parallel corpora consisting of English and German, parallel corpora consisting of English and Spanish, and parallel corpora consisting of English and Chinese.

[0016] Furthermore, the multilingual data set is preprocessed as follows:

[0017] For Chinese corpus: After using the jieba tool to standardize the punctuation marks of the Chinese corpus, the Chinese corpus is segmented.

[0018] For other corpora: the tool libraries Moses and SubwordNMT are used to tokenize, convert uppercase to lowercase, and clean the corpus.

[0019] Furthermore, the multilingual global dictionary is constructed based on the preprocessing results, specifically:

[0020] Step S101: using a subword segmentation method to segment the word segmentation results of each language respectively, so that the number of words obtained after segmentation in each language reaches a preset vocabulary size;

[0021] Step S102: Merge the vocabulary lists of various languages ​​into a multilingual global shared dictionary.

[0022] Furthermore, the multilingual neural machine translation model is pre-trained based on the meta-learning method, the multilingual global shared dictionary, and the training set, the validation set, and the test set to obtain a pre-trained multilingual neural machine translation model; the specific process is:

[0023] Step S201: Divide the training set into a support set and a query set, and establish a total task distribution p(τ) that includes the probability of each translation task occurring during the pre-training process;

[0024] Step S202: Set the inner learning rate to α and the outer learning rate to β; randomly initialize the parameters θ0 of the multilingual neural machine translation model;

[0025] Step S203, initialize the number of pre-training rounds n=1;

[0026] Step S204: Initialize the number of non-cross-task pre-training times of the current round T=1;

[0027] Step S205: Randomly select the T-th pre-trained translation task τ from the total task distribution p(τ) T , from the translation task τ T A batch of language pair data is randomly selected from the corresponding data in the support set. The preprocessing results corresponding to the selected language pair data are used as the input of the multilingual neural machine translation model. The multilingual neural machine translation model outputs translation results based on the multilingual global shared dictionary. The cross-entropy loss function is calculated based on the translation results output by the multilingual neural machine translation model.

[0028] Calculate the gradient based on the cross entropy loss function, and then update the parameters of the multilingual neural machine translation model based on the gradient to obtain the model parameters after the Tth update;

[0029] Step S206: determine whether T=T′ is satisfied, where T′ is the maximum number of non-cross-task pre-training times set;

[0030] If T=T′ is not satisfied, set T=T+1 and return to step S205;

[0031] If T=T′, continue to step S207;

[0032] Step S207: Use the preprocessing results of the multi-language pairs in the query set to test the T′ group of model parameters obtained from the nth round of non-cross-task pre-training. The T′ group of model parameters all output translation results based on the multi-language global shared dictionary. Then, based on the translation results output by the T′ group of model parameters, update the model parameters after the (n-1)th round of pre-training to obtain the model parameters after the nth round of pre-training.

[0033] Step S208: Determine whether n=N is satisfied, where N represents the maximum number of pre-training rounds set;

[0034] If n=N is not satisfied, n=n+1 is set and the process returns to step S204;

[0035] If n=N, continue to execute step S209;

[0036] Step S209: Use the data in the validation set to verify the training effect of the model parameters after the Nth round of pre-training. If the verification passes, execute step S210. If the verification fails, return to step S203 and continue pre-training the model.

[0037] Step S210: Use the data in the test set to test the verified model. If the test passes, a pre-trained multilingual neural machine translation model is obtained. If the test fails, return to step S203 to continue pre-training the model.

[0038] Furthermore, in step S205, the parameters of the multilingual neural machine translation model are updated according to the gradient; specifically:

[0039]

[0040] Among them, θ n-1 is the model parameter after the n-1th round of pre-training, θ T ' is the model parameter after the Tth non-cross-task pre-training update, is the cross entropy loss function, yes gradient.

[0041] Furthermore, in step S207, the model parameters after the n-1th round of pre-training are updated according to the translation results output by the T′ group of model parameters to obtain the model parameters after the nth round of pre-training; specifically,

[0042]

[0043] Among them, θ n are the model parameters after the nth round of pre-training, is the loss of the model parameters of the multi-language pairs in the query set after the T-th non-cross-task pre-training update, express gradient.

[0044] Furthermore, the multilingual neural machine translation model adopts a Multilingual Transformer model.

[0045] Furthermore, for any language pair in the multi-language pair dataset, the parameters of the subnet after the parameters are loaded are fine-tuned using the preprocessing results corresponding to the language pair, and a mask matrix for the language pair is generated according to the fine-tuning results. The specific process is as follows:

[0046] Step S301: record the data set composed of the language pair as Among them, D sj→tj represents the j-th language pair in the language pair dataset, N represents the number of language pairs in the language pair dataset, sj represents the source language in the language pair, and tj represents the target language in the language pair;

[0047] Step S302: fine-tune the parameters of the subnet after the parameters are loaded using the pre-processing result corresponding to the data set in the language;

[0048] Step S303: Prune the model parameters according to the size of the fine-tuned model parameter weights. The layers whose parameters are pruned include the source language word embedding layer, the multi-head attention layer in the encoder, the feedforward neural network layer, the multi-head attention layer in the decoder, and the feedforward neural network layer. The pruning process is as follows:

[0049] For any layer in the subnet, all parameters in the layer are sorted in descending order according to the parameter weight, and the parameters with the top α% weight are selected for the layer, that is, the parameters with 1-α% weight are pruned. In the mask vector corresponding to the layer, the values ​​corresponding to the selected parameters are recorded as 1, and the values ​​corresponding to the pruned parameters are recorded as 0;

[0050] Similarly, the mask vector corresponding to each layer in the subnet is obtained respectively; the mask vector corresponding to each layer is used to form the mask matrix of the subnet.

[0051] Furthermore, the specific process of step S302 is as follows:

[0052]

[0053] in, is the loss function, <x,y> is the Dth sj→tj A sentence pair consisting of a source language corresponding sentence and a target language corresponding sentence in a language pair, Given a source language sentence x, the model has parameters The probability of generating the target language sentence y under

[0054] The beneficial effects of the present invention are:

[0055] 1. This paper uses ML methods to optimize the initial parameters of a multilingual translation model, achieving a local optimal parameter value. By learning shared representations across tasks and adaptively adjusting each language pair, ML can reduce parameter interference, improve multilingual translation quality, and ensure translation performance.

[0056] 2. This paper uses ML methods to learn cross-lingual knowledge representation between high-resource language pairs and low-resource language pairs, enabling the model to quickly adapt to different language pair tasks.

[0057] 3. By learning language-specific subnetworks, this invention can obtain a mask matrix for each language pair task. During model training, only the model parameters corresponding to the input language pair data are updated at a time. This allows multilingual translation models to avoid parameter interference and improve training efficiency. In practical application, for a specific translation task, only the model parameters corresponding to element 1 in the mask matrix corresponding to the translation task are used to predict the translation result, reducing the complexity of the model calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a multilingual neural machine translation model training method based on meta-learning and language pair subnets of the present invention;

[0059] Figure 2 This is a meta-path diagram of four English-centered translation tasks;

[0060] Figure 3 This is a flowchart of the meta-learning training process;

[0061] Figure 4 Schematic diagram of the multilingual neural machine translation model and language pair subnet structure. DETAILED DESCRIPTION

[0062] Specific implementation method 1: Combination Figure 1 This embodiment describes a method for training a multilingual neural machine translation model based on meta-learning and language pair subnets, the method specifically comprising the following steps:

[0063] Step 1: Obtain a multi-language pair dataset, preprocess the multi-language pair dataset, and build a multi-language global shared dictionary based on the pre-processed multi-language pairs. After adding a language category identifier to the front of the pre-processed multi-language pairs, the pre-processed multi-language pairs are divided into three parts: training set, validation set, and test set (the division ratio is: each language pair dataset is divided into 80% training set, 10% validation set, and 10% test set. For example, in the French-English dataset, 24,000 items are divided into training set, 3,000 items into validation set, and 3,000 items into test set).

[0064] Step 2: Pre-train the multilingual neural machine translation model based on the meta-learning method, the multilingual global shared dictionary, and the training set, validation set, and test set to obtain a pre-trained multilingual neural machine translation model;

[0065] Step 3: Build a subnet for each language pair in the multilingual dataset. The structure of the subnet for each language pair is the same as that of the multilingual neural machine translation model. Each subnet is loaded with the parameters of the pre-trained multilingual neural machine translation model.

[0066] For any language pair in the multi-language dataset, fine-tune the parameters of the subnet after loading the parameters using the preprocessing results corresponding to the language pair, and generate the mask matrix of the language pair based on the fine-tuning results;

[0067] Step 4: Use the preprocessed results corresponding to each language pair to train the pre-trained multilingual neural machine translation model to obtain a trained multilingual neural machine translation model;

[0068] When the pre-trained model is trained using the pre-processed results corresponding to the i-th language pair, only the parameter weight corresponding to element 1 in the mask matrix corresponding to the i-th language pair is adjusted;

[0069] Step 5: After preprocessing the sentence to be translated, add a language category identifier to the preprocessing result, and then use the preprocessing result with the added language category identifier as the input of the trained multilingual neural machine translation model, and output the translation result through the trained multilingual neural machine translation model.

[0070] Specific implementation method 2: Combination Figure 2 This embodiment differs from the first embodiment in that the multi-language pair dataset is downloaded from IWSLT14 and specifically includes a parallel corpus consisting of English and French, a parallel corpus consisting of English and German, a parallel corpus consisting of English and Spanish, and a parallel corpus consisting of English and Chinese.

[0071] Other steps and parameters are the same as those in the first embodiment.

[0072] The parallel corpora downloaded in the present invention may include but are not limited to the aforementioned parallel corpora.

[0073] Specific embodiment three: This embodiment differs from specific embodiments one or two in that the multilingual data set is preprocessed as follows:

[0074] For Chinese corpus: After using the jieba tool to standardize the punctuation marks of the Chinese corpus, the Chinese corpus is segmented.

[0075] For other corpora: the tool libraries Moses and SubwordNMT are used to tokenize the corpus (equivalent to word segmentation), convert uppercase to lowercase, and perform cleaning operations.

[0076] Other steps and parameters are the same as those in the first or second embodiment.

[0077] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the multilingual global dictionary is constructed based on the preprocessing results, specifically:

[0078] Step S101: Segment the word segmentation results of each language using a word segmentation method (BPE) so that the number of words obtained after segmentation in each language reaches a preset vocabulary size (the specific size of the vocabulary can be set according to actual conditions);

[0079] Step S102: Merge the vocabulary lists of various languages ​​into a multilingual global shared dictionary.

[0080] The other steps and parameters are the same as those in the first to third embodiments.

[0081] Specific implementation method five: Combination Figure 3 This embodiment differs from Embodiments 1 to 4 in that the multilingual neural machine translation model is pre-trained based on a meta-learning method, a multilingual global shared dictionary, and a training set, a validation set, and a test set to obtain a pre-trained multilingual neural machine translation model. The specific process is as follows:

[0082] Step S201: Divide the training set into a support set and a query set, and establish a total task distribution p(τ) that includes the probability of each translation task occurring during the pre-training process. The probability of each translation task occurring during the pre-training process can be manually designed. By designing a balanced probability, the model can quickly adapt to high-resource English-French, English-Spanish, English-Chinese and low-resource English-German language tasks, and is not affected by the data imbalance between high-resource and low-resource languages, thereby improving the translation effect;

[0083] Step S202: Set the inner learning rate to α and the outer learning rate to β; randomly initialize the parameters θ0 of the multilingual neural machine translation model;

[0084] Step S203, initialize the number of pre-training rounds n=1;

[0085] Step S204: Initialize the number of non-cross-task pre-training times of the current round T=1;

[0086] Step S205: Randomly select the T-th pre-trained translation task τ from the total task distribution p(τ) T (A translation task corresponds to only one type of language pair), from the translation task τ T A batch of language pairs is randomly selected from the corresponding data in the support set. The preprocessing results corresponding to the selected language pairs (the preprocessing results input into the model are all preprocessing results after adding language category identifiers) are used as the input of the multilingual neural machine translation model. The multilingual neural machine translation model outputs translation results based on the multilingual global shared dictionary. The cross-entropy loss function is calculated based on the translation results output by the multilingual neural machine translation model.

[0087] Calculate the gradient based on the cross entropy loss function, and then update the parameters of the multilingual neural machine translation model based on the gradient to obtain the model parameters after the Tth update;

[0088] Step S206: determine whether T=T′ is satisfied, where T′ is the maximum number of non-cross-task pre-training times set;

[0089] If T=T′ is not satisfied, set T=T+1 and return to step S205;

[0090] If T=T′, continue to step S207;

[0091] Step S207: Use the preprocessing results of the multi-language pairs in the query set to test the T′ group of model parameters obtained from the nth round of non-cross-task pre-training. The T′ group of model parameters all output translation results based on the multi-language global shared dictionary. Then, based on the translation results output by the T′ group of model parameters, update the model parameters after the (n-1)th round of pre-training to obtain the model parameters after the nth round of pre-training.

[0092] Step S208: Determine whether n=N is satisfied, where N represents the maximum number of pre-training rounds set;

[0093] If n=N is not satisfied, n=n+1 is set and the process returns to step S204;

[0094] If n=N, continue to execute step S209;

[0095] Step S209: Use the data in the validation set to verify the training effect of the model parameters after the Nth round of pre-training. If the verification passes (i.e., the translation effect of the model on the validation set meets the requirements), proceed to step S210. If the verification fails, return to step S203 and continue pre-training the model.

[0096] Step S210: Use the data in the test set to test the verified model. If the test passes (that is, the translation effect of the model on the test set meets the requirements), a pre-trained multilingual neural machine translation model is obtained. If the test fails, return to step S203 and continue pre-training the model.

[0097] The other steps and parameters are the same as those in the first to fourth embodiments.

[0098] The present invention optimizes the model across tasks, enabling the model to quickly adapt to new language pairs with a small amount of data and gradient updates.

[0099] Specific embodiment 6: This embodiment differs from any one of specific embodiments 1 to 5 in that, in step S205, the parameters of the multilingual neural machine translation model are updated according to the gradient; specifically:

[0100]

[0101] Among them, θ n-1 is the model parameter after the n-1th round of pre-training (for the first round of pre-training, when n=1, θ n-1 =θ0, for the second and subsequent rounds of pre-training, θ n-1 is the model parameter obtained in the previous round of pre-training), θ T ' is the model parameter after the Tth non-cross-task pre-training update, is the cross entropy loss function, yes gradient.

[0102] The other steps and parameters are the same as those in the first to fifth embodiments.

[0103] It should be noted that in the same round of pre-training, each non-cross-task pre-training is to update the network parameters obtained in the previous round of pre-training.

[0104] Specific embodiment 7: This embodiment differs from any one of specific embodiments 1 to 6 in that, in step S207, the model parameters after the n-1th round of pre-training are updated according to the translation results output by the T′ group of model parameters to obtain the model parameters after the nth round of pre-training; specifically,

[0105]

[0106] Among them, θ n are the model parameters after the nth round of pre-training, is the loss of the model parameters of the multi-language pairs in the query set after the T-th non-cross-task pre-training update, express gradient.

[0107] The other steps and parameters are the same as those in the first to sixth embodiments.

[0108] The T′ groups of model parameters are obtained through pre-training based on various types of language pairs. For any type of language pair, data for that language pair is obtained from the query set. The obtained data and the groups of model parameters obtained based on that language pair are then used to obtain groups of translation results for the obtained data. The loss is then calculated based on the obtained translation results.

[0109] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that the multilingual neural machine translation model adopts a Multilingual Transformer model.

[0110] The other steps and parameters are the same as those in the first to seventh embodiments.

[0111] The basic architecture of the Multilingual Transformer model is similar to that of the standard Transformer model. In the encoder, a language tag is added to the source language to inform the model of the language of the sentence. In the decoder, a language tag is also added to the target language to help the model distinguish the target language of the translation. The encoder of the Multilingual Transformer model encodes the input sequence into a hidden representation, and the decoder decodes the hidden representation into the target sequence. The encoder is composed of multiple identical encoder layers stacked together. Each encoder layer contains two sublayers: a multi-head self-attention mechanism and a fully connected feedforward neural network. The decoder is also composed of multiple identical decoder layers stacked together. Each decoder layer has an additional multi-head masked self-attention sublayer compared to the encoder layer. Residual connections and layer regularization are applied after each sublayer. Residual connections help alleviate the vanishing and exploding gradient problems, while layer regularization helps accelerate model convergence.

[0112] Specific implementation method nine: Combination Figure 4This embodiment differs from Embodiments 1 to 8 in that, for any language pair in the multi-language dataset, the preprocessing results corresponding to that language pair are used to fine-tune the parameters of the subnet after the parameters are loaded, and a mask matrix for that language pair is generated based on the fine-tuning results. The specific process is as follows:

[0113] Step S301: record the data set composed of the language pair as Among them, D sj→tj represents the j-th language pair in the language pair dataset, N represents the number of language pairs in the language pair dataset, sj represents the source language in the language pair, and tj represents the target language in the language pair;

[0114] Step S302: fine-tune the parameters of the subnet after the parameters are loaded using the pre-processing result corresponding to the data set in the language;

[0115] Step S303: Prune the model parameters according to the size of the fine-tuned model parameter weights. The layers whose parameters are pruned include the source language word embedding layer, the multi-head attention layer in the encoder, the feedforward neural network layer, the multi-head attention layer in the decoder, and the feedforward neural network layer. The pruning process is as follows:

[0116] For any layer in the subnet, after sorting all the parameters in the layer in descending order of parameter weight, select the parameters with the top α% weight for the layer (the pruning ratio is 70% for high-resource languages ​​(English-French, English-Spanish, English-Chinese), and the pruning ratio is 80% for low-resource languages ​​​​English-German). That is, 1-α% of the parameters are pruned, and the values ​​corresponding to the selected parameters are recorded as 1 in the mask vector corresponding to the layer, and the values ​​corresponding to the pruned parameters are recorded as 0;

[0117] Similarly, the mask vector corresponding to each layer in the subnet is obtained respectively; the mask vector corresponding to each layer is used to form the mask matrix of the subnet.

[0118] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0119] The subnetwork for each language pair is optimized for the specific tasks of that language pair, so it can better handle the translation tasks of that language pair without being interfered with by other language pairs. The generated mask matrix is ​​actually a matrix of the same size as the model parameters. Each element in the mask matrix indicates whether each parameter is retained. The mask matrix is ​​associated with the subnetwork corresponding to each language pair. The mask matrix of the subnetwork retains some shared parameters, while maintaining specific weights for each language pair, thereby reducing interference while retaining the generalization ability of the multilingual model. In subsequent training, only the weights of the parameters corresponding to element 1 in the mask matrix are updated, and the weights of the parameters corresponding to element 0 in the mask matrix are not involved in training. This process is called structural perception.

[0120] The complete network and language-specific subnets are Figure 4 As shown in Figure 2, parameter interference between different language pairs is avoided. Through fine-tuning, pruning, and structure-aware training, the final multilingual translation model is obtained, further improving the quality of multilingual translation and alleviating the poor performance of low-resource languages ​​in multilingual translation.

[0121] Specific embodiment 10: This embodiment differs from any one of specific embodiments 1 to 9 in that the specific process of step S302 is as follows:

[0122]

[0123] in, is the loss function, <x,y> is the Dth sj→tj A sentence pair consisting of a source language corresponding sentence and a target language corresponding sentence in a language pair, Given a source language sentence x, the model has parameters The probability of generating the target language sentence y under

[0124] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0125] The base of the logarithm in the present invention may be 10, but is not limited to 10. This fine-tuning method will increase the weight of important parameters for the type of language, while reducing the weight of unimportant parameters.

[0126] After obtaining the masks for each language pair, the pre-trained model obtained based on the ML method is retrained, and the model θ is trained using batching and structure-aware updates grouped by language pairs. The batching of language pairs creates random batches, and each batch contains only parallel sentences from a single language pair. For example, batch B sj→tj Contains only sentence pairs D from the language pair sj→tj sj→tj , for each language pair, batch B sj→tj Calculate the loss function, the loss function is as follows:

[0127]

[0128] Among them, P θ (y|x) is the probability that the model generates the target language sentence y under the parameters θ given the source language sentence x. The goal is to minimize this loss, that is, to maximize the probability of generating the correct target sentence. During the backpropagation process, only the model parameters related to the current task are updated (that is, the parameters corresponding to element 1 in the mask matrix of the current input language pair). Specifically, only the mask matrix M sj→tj The indicated parameters are updated to ensure that the parameters important for each language pair are optimized. The model parameters are updated through multiple iterations until training converges. In this way, the parameters important for each language pair are optimized based on the training data for that language pair. Ultimately, training results in a unified model θ (containing both shared and specialized parameters) that can handle all language pairs and accomplish multilingual translation tasks.

[0129] Through the above-mentioned fine-tuning, pruning and structure-aware training, a model suitable for a variety of specific language pairs can be obtained. Specifically, using θ * Represents the final model parameters after training, which are used in the inference phase to generate prediction results. In the prediction phase, the mask matrix M sj→tj It is applied to the final model parameters, and according to the mask matrix, the model parameters are selected for a specific source language to target language pair, and the selected model parameters are used in the inference process.

[0130] like Figure 4 As shown, during inference, the final model parameter θ * Will be masked M sj→tj Filtering, so that the model only uses the network parameters corresponding to a specific language pair to generate translation results.

[0131] By introducing a specific mask matrix for each language pair, we address the parameter interference issue in multilingual translation model training and improve the translation quality of multiple language pairs. Furthermore, through fine-tuning, pruning, and structure-aware training, each language pair receives a dedicated parameter subset. This enables the model to handle multiple language pairs while effectively improving translation performance for each language pair.

[0132] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A multilingual neural machine translation model training method based on meta-learning and language pair subnet, characterized in that: The method specifically comprises the following steps: Step 1: Obtain a multi-language dataset, preprocess it, and build a multi-language global shared dictionary based on the pre-processed multi-language pairs. Add a language category identifier to the pre-processed multi-language pairs, and then divide the pre-processed multi-language pairs into three parts: training set, validation set, and test set. Step 2: Pre-train the multilingual neural machine translation model based on the meta-learning method, the multilingual global shared dictionary, and the training set, validation set, and test set to obtain a pre-trained multilingual neural machine translation model; Step 3: Build a subnet for each language pair in the multilingual dataset. The structure of the subnet for each language pair is the same as that of the multilingual neural machine translation model. Each subnet is loaded with the parameters of the pre-trained multilingual neural machine translation model. For any language pair in the multi-language dataset, fine-tune the parameters of the subnet after loading the parameters using the preprocessing results corresponding to the language pair, and generate the mask matrix of the language pair based on the fine-tuning results; Step 4: Use the preprocessed results corresponding to each language pair to train the pre-trained multilingual neural machine translation model to obtain a trained multilingual neural machine translation model; When the pre-trained model is trained using the pre-processed results corresponding to the i-th language pair, only the parameter weight corresponding to element 1 in the mask matrix corresponding to the i-th language pair is adjusted; Step 5: After preprocessing the sentence to be translated, add a language category identifier to the preprocessing result, and then use the preprocessing result with the added language category identifier as the input of the trained multilingual neural machine translation model, and output the translation result through the trained multilingual neural machine translation model.

2. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnet according to claim 1, characterized in that: The multi-language pair dataset is downloaded from IWSLT14, and specifically includes parallel corpora consisting of English and French, parallel corpora consisting of English and German, parallel corpora consisting of English and Spanish, and parallel corpora consisting of English and Chinese.

3. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnet according to claim 2, characterized in that: The multilingual dataset is preprocessed as follows: For Chinese corpus: After using the jieba tool to standardize the punctuation marks of the Chinese corpus, the Chinese corpus is segmented. For other corpora: the tool libraries Moses and SubwordNMT are used to tokenize, convert uppercase to lowercase, and clean the corpus.

4. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnets according to claim 3, characterized in that: The multilingual global dictionary is constructed based on the preprocessing results, specifically: Step S101: using a subword segmentation method to segment the word segmentation results of each language respectively, so that the number of words obtained after segmentation in each language reaches a preset vocabulary size; Step S102: Merge the vocabulary lists of various languages ​​into a multilingual global shared dictionary.

5. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnet according to claim 4, characterized in that: The multilingual neural machine translation model is pre-trained based on the meta-learning method, the multilingual global shared dictionary, and the training set, the validation set, and the test set to obtain a pre-trained multilingual neural machine translation model. The specific process is as follows: Step S201: Divide the training set into a support set and a query set, and establish a total task distribution p(τ) that includes the probability of each translation task occurring during the pre-training process; Step S202: Set the inner learning rate to α and the outer learning rate to β; randomly initialize the parameters θ0 of the multilingual neural machine translation model; Step S203, initialize the number of pre-training rounds n=1; Step S204: Initialize the number of non-cross-task pre-training times of the current round T=1; Step S205: Randomly select the T-th pre-trained translation task τ from the total task distribution p(τ) T , from the translation task τ T A batch of language pair data is randomly selected from the corresponding data in the support set. The preprocessing results corresponding to the selected language pair data are used as the input of the multilingual neural machine translation model. The multilingual neural machine translation model outputs translation results based on the multilingual global shared dictionary. The cross-entropy loss function is calculated based on the translation results output by the multilingual neural machine translation model. Calculate the gradient based on the cross entropy loss function, and then update the parameters of the multilingual neural machine translation model based on the gradient to obtain the model parameters after the Tth update; Step S206: determine whether T=T′ is satisfied, where T′ is the maximum number of non-cross-task pre-training times set; If T=T′ is not satisfied, set T=T+1 and return to step S205; If T=T′, continue to step S207; Step S207: Use the preprocessing results of the multi-language pairs in the query set to test the T′ group of model parameters obtained from the nth round of non-cross-task pre-training. The T′ group of model parameters all output translation results based on the multi-language global shared dictionary. Then, based on the translation results output by the T′ group of model parameters, update the model parameters after the (n-1)th round of pre-training to obtain the model parameters after the nth round of pre-training. Step S208: Determine whether n=N is satisfied, where N represents the maximum number of pre-training rounds set; If n=N is not satisfied, n=n+1 is set and the process returns to step S204; If n=N, continue to execute step S209; Step S209: Use the data in the validation set to verify the training effect of the model parameters after the Nth round of pre-training. If the verification passes, execute step S210. If the verification fails, return to step S203 and continue pre-training the model. Step S210: Use the data in the test set to test the verified model. If the test passes, a pre-trained multilingual neural machine translation model is obtained. If the test fails, return to step S203 to continue pre-training the model.

6. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnets according to claim 5, characterized in that: In step S205, the parameters of the multilingual neural machine translation model are updated according to the gradient; specifically: Among them, θ n-1 is the model parameter after the n-1th round of pre-training, θ T ' is the model parameter after the Tth non-cross-task pre-training update, is the cross entropy loss function, yes gradient.

7. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnets according to claim 6, characterized in that: In step S207, the model parameters after the n-1th round of pre-training are updated according to the translation results output by the T′ group of model parameters to obtain the model parameters after the nth round of pre-training; specifically, Among them, θ n are the model parameters after the nth round of pre-training, is the loss of the model parameters of the multi-language pairs in the query set after the T-th non-cross-task pre-training update, express gradient.

8. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnets according to claim 7, characterized in that: The multilingual neural machine translation model adopts the Multilingual Transformer model.

9. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnets according to claim 8, characterized in that: For any language pair in the multi-language pair dataset, fine-tune the parameters of the subnet after the parameters are loaded using the preprocessing results corresponding to the language pair, and generate a mask matrix for the language pair according to the fine-tuning results; The specific process is: Step S301: record the data set composed of the language pair as Among them, D sj→tj represents the j-th language pair in the language pair dataset, N represents the number of language pairs in the language pair dataset, sj represents the source language in the language pair, and tj represents the target language in the language pair; Step S302: fine-tune the parameters of the subnet after the parameters are loaded using the pre-processing result corresponding to the data set in the language; Step S303: Prune the model parameters according to the size of the fine-tuned model parameter weights. The layers whose parameters are pruned include the source language word embedding layer, the multi-head attention layer in the encoder, the feedforward neural network layer, the multi-head attention layer in the decoder, and the feedforward neural network layer. The pruning process is as follows: For any layer in the subnet, all parameters in the layer are sorted in descending order according to the parameter weight, and the parameters with the top α% weight are selected for the layer, that is, the parameters with 1-α% weight are pruned. In the mask vector corresponding to the layer, the values ​​corresponding to the selected parameters are recorded as 1, and the values ​​corresponding to the pruned parameters are recorded as 0; Similarly, the mask vector corresponding to each layer in the subnet is obtained respectively; the mask vector corresponding to each layer is used to form the mask matrix of the subnet.

10. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnets according to claim 9, characterized in that: The specific process of step S302 is: in, is the loss function, <x,y> is the Dth sj→tj A sentence pair consisting of a source language corresponding sentence and a target language corresponding sentence in a language pair, Given a source language sentence x, the model has parameters The probability of generating the target language sentence y under

Images