Multi-language neural machine translation model training method based on meta learning and language pair subnet

Abstract

The invention discloses a multi-language neural machine translation model training method based on meta learning and language pair subnets, and belongs to the technical field of machine translation. The problem that an existing multi-language translation sharing model is poor in translation performance is solved. According to the method, initial parameter optimization is carried out on a multi-language translation model by means of an ML method to obtain a local optimal parameter value, and the ML learns sharing representation among tasks and carries out adaptive adjustment on each language pair, so that parameter interference can be reduced, and translation performance is ensured. A mask matrix of each language to a task can be obtained by learning a language specific subnet, and only model parameters corresponding to input language to data are updated each time during model training, so that a multi-language translation model can avoid parameter interference, and meanwhile, the training efficiency is improved. In the subsequent translation task, only the model parameter corresponding to the element 1 in the mask matrix is needed to participate in translation result prediction, so that the complexity of model calculation is reduced. The method can be applied to the field of machine translation.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation, and in particular relates to a multilingual neural machine translation model training method based on meta-learning (ML) and language pair subnets. Background Art

[0002] Due to the wide variety of languages ​​used by humans, language communication barriers are very likely to occur for people who are not familiar with multiple languages. With the development of deep learning, traditional neural machine translation (NMT) methods usually train independent translation models for each pair of languages ​​(source language and target language). Each model independently learns the mapping relationship between the source language and the target language. Although this method can provide high-quality translation results, the computational and storage costs will rise sharply as the number of language pairs increases. Especially in the case of training multiple language pairs, redundant computation and storage make model expansion difficult. Multilingual neural machine translation (MNMT) processes multiple language pairs by sharing models, reducing the computational burden of independent training for each pair of languages ​​in traditional NMT methods. In multilingual translation tasks, multiple language pairs share a translation model, but as the number of tasks increases, the model parameters will become complicated (it is impossible to consider the relationship between tasks and the specific situation within the task).

[0003] Therefore, the existing multilingual translation sharing model still has the following problems:

[0004] (1) The differences in tasks between different language pairs lead to parameter interference in the multilingual translation sharing model, that is, the parameter optimization of some language pairs will affect the translation effect of other language pairs (especially the data imbalance between high-resource and low-resource languages).

[0005] (2) There are redundant parameters in the multi-language translation sharing model, which impairs the performance of single-language translation. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of poor translation performance of the existing multilingual translation sharing model, and propose a multilingual neural machine translation model training method based on meta-learning and language pair subnets.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a multilingual neural machine translation model training method based on meta-learning and language pair subnet, the method specifically comprises the following steps:

[0008] Step 1: obtain a multi-language pair dataset, pre-process the multi-language pair dataset, and build a multi-language global shared dictionary based on the pre-processed multi-language pairs; add a language category identifier in front of the pre-processed multi-language pairs, and then divide the pre-processed multi-language pairs into three parts: a training set, a validation set, and a test set;

[0009] Step 2: Pre-train the multilingual neural machine translation model based on the meta-learning method, the multilingual global shared dictionary, and the training set, the validation set, and the test set to obtain a pre-trained multilingual neural machine translation model;

[0010] Step 3: Build a subnet for each language pair in the multi-language pair dataset, and the structure of the subnet for each language pair is the same as that of the multi-language neural machine translation model. Each subnet is loaded with the parameters of the pre-trained multi-language neural machine translation model.

[0011] For any language pair in the multi-language pair dataset, fine-tune the parameters of the subnet after loading the parameters using the preprocessing results corresponding to the language pair, and generate a mask matrix for the language pair based on the fine-tuning results;

[0012] Step 4: Use the preprocessed results corresponding to each language pair to train the pre-trained multilingual neural machine translation model to obtain a trained multilingual neural machine translation model;

[0013] When the pre-trained model is trained using the pre-processed result corresponding to the i-th language pair, only the parameter weight corresponding to element 1 in the mask matrix corresponding to the i-th language pair is adjusted;

[0014] Step 5: After preprocessing the sentence to be translated, add a language category identifier to the preprocessing result, and then use the preprocessing result with the language category identifier as the input of the trained multilingual neural machine translation model, and output the translation result through the trained multilingual neural machine translation model.

[0015] Furthermore, the multi-language pair dataset is downloaded from IWSLT14, and the multi-language pair dataset specifically includes a parallel corpus consisting of English and French, a parallel corpus consisting of English and German, a parallel corpus consisting of English and Spanish, and a parallel corpus consisting of English and Chinese.

[0016] Furthermore, the multilingual data set is preprocessed as follows:

[0017] For Chinese corpus: After using the jieba tool to standardize the punctuation marks of the Chinese corpus, the Chinese corpus is segmented.

[0018] For other corpora: the tool libraries Moses and SubwordNMT are used to tokenize, convert uppercase to lowercase, and clean the corpus.

[0019] Furthermore, the multilingual global dictionary is constructed according to the preprocessing results, specifically:

[0020] Step S101, using a subword segmentation method to segment the word segmentation results of each language respectively, so that the number of words obtained after segmentation in each language reaches a preset vocabulary size;

[0021] Step S102: Merge the vocabulary tables of various languages ​​into a multi-language global shared dictionary.

[0022] Furthermore, the multilingual neural machine translation model is pre-trained based on the meta-learning method, the multilingual global shared dictionary, and the training set, the validation set, and the test set to obtain a pre-trained multilingual neural machine translation model; the specific process is:

[0023] Step S201, dividing the training set into a support set and a query set, and establishing a total task distribution p(τ) including the probability of each translation task occurring during the pre-training process;

[0024] Step S202, setting the inner layer learning rate to α and the outer layer learning rate to β; randomly initializing the parameter θ0 of the multilingual neural machine translation model;

[0025] Step S203, initializing the number of pre-training rounds n=1;

[0026] Step S204, initializing the number of non-cross-task pre-training times of the current round T=1;

[0027] Step S205: Randomly select the T-th pre-trained translation task τ from the total task distribution p(τ) T , from the translation task τ T A batch of language pair data is randomly selected from the corresponding data in the support set, and the preprocessing results corresponding to the selected language pair data are used as the input of the multilingual neural machine translation model. The multilingual neural machine translation model outputs the translation results according to the multilingual global shared dictionary, and the cross entropy loss function is calculated according to the translation results output by the multilingual neural machine translation model;

[0028] Calculate the gradient according to the cross entropy loss function, and then update the parameters of the multilingual neural machine translation model according to the gradient to obtain the model parameters after the Tth update;

[0029] Step S206, determining whether T=T′ is satisfied, where T′ is the maximum number of non-cross-task pre-training times set;

[0030] If T=T′ is not satisfied, then set T=T+1 and return to step S205;

[0031] If T=T′ is satisfied, then continue to execute step S207;

[0032] Step S207: Use the preprocessing results of the multi-language pairs in the query set to test the T′ group of model parameters obtained by the nth round of non-cross-task pre-training. The T′ group of model parameters all output translation results according to the multi-language global shared dictionary. Then, according to the translation results output by the T′ group of model parameters, update the model parameters after the n-1th round of pre-training to obtain the model parameters after the nth round of pre-training.

[0033] Step S208: determine whether n=N is satisfied, where N represents the maximum number of pre-training rounds set;

[0034] If n=N is not satisfied, n=n+1 is set and the process returns to step S204;

[0035] If n=N, continue to execute step S209;

[0036] Step S209: Use the data in the validation set to verify the training effect of the model parameters after the Nth round of pre-training. If the verification passes, execute step S210; if the verification fails, return to step S203 to continue pre-training the model.

[0037] Step S210: Use the data in the test set to test the verified model. If the test passes, a pre-trained multilingual neural machine translation model is obtained. If the test fails, return to step S203 to continue pre-training the model.

[0038] Furthermore, in step S205, the parameters of the multilingual neural machine translation model are updated according to the gradient; specifically:

[0039]

[0040] Among them, θ n-1 is the model parameter after the n-1th round of pre-training, θ T ' is the model parameter after the Tth non-cross-task pre-training update, is the cross entropy loss function, yes gradient.

[0041] Furthermore, in step S207, the model parameters after the n-1th round of pre-training are updated according to the translation results output by the T′ group of model parameters to obtain the model parameters after the nth round of pre-training; specifically:

[0042]

[0043] Among them, θ n are the model parameters after the nth round of pre-training, is the loss of the model parameters of the multi-language pairs in the query set after the T-th non-cross-task pre-training update, express gradient.

[0044] Furthermore, the multilingual neural machine translation model adopts a Multilingual Transformer model.

[0045] Furthermore, for any language pair in the multi-language pair dataset, the parameters of the subnet after the parameters are loaded are fine-tuned using the preprocessing result corresponding to the language pair, and a mask matrix of the language pair is generated according to the fine-tuning result; the specific process is:

[0046] Step S301: record the data set consisting of the language pair as Among them, D sj→tj represents the jth language pair in the language pair dataset, N represents the number of language pairs in the language pair dataset, sj represents the source language in the language pair, and tj represents the target language in the language pair;

[0047] Step S302, using the language to fine-tune the parameters of the subnet after the parameters are loaded, based on the preprocessing result corresponding to the data set;

[0048] Step S303: prune the model parameters according to the size of the fine-tuned model parameter weights. The layers for parameter pruning specifically include the source language word embedding layer, the multi-head attention layer in the encoder, the feedforward neural network layer, the multi-head attention layer in the decoder, and the feedforward neural network layer. The pruning process is specifically as follows:

[0049] For any layer in the subnet, after sorting all the parameters in the layer in descending order according to the parameter weight, select the parameters with the top α% weight for the layer, that is, prune the parameters of 1-α%, and in the mask vector corresponding to the layer, record the value corresponding to the selected parameter as 1, and the value corresponding to the pruned parameter as 0;

[0050] Similarly, the mask vectors corresponding to each layer in the subnet are obtained respectively; and the mask matrix of the subnet is composed using the mask vectors corresponding to each layer.

[0051] Furthermore, the specific process of step S302 is as follows:

[0052]

[0053] in, is the loss function, 〈x,y〉 is the Dth sj→tj A sentence pair consisting of a source language corresponding sentence and a target language corresponding sentence in a language pair. Given a source language sentence x, the model has parameters The probability of generating the target language sentence y under

[0054] The beneficial effects of the present invention are:

[0055] 1. The present invention uses the ML method to optimize the initial parameters of the multilingual translation model and obtain a local optimal parameter value. ML can reduce parameter interference, improve the quality of multilingual translation, and ensure translation performance by sharing representations between learning tasks and adaptively adjusting each language pair.

[0056] 2. The present invention uses ML methods to learn cross-language knowledge representation between high-resource language pairs and low-resource language pairs, so that the model can quickly adapt to different language pair tasks.

[0057] 3. The present invention can obtain the mask matrix of each language pair task by learning the language-specific subnet. During model training, only the model parameters corresponding to the input language pair data are updated each time, so that the multilingual translation model can avoid parameter interference and improve training efficiency. When the model is actually applied, for a specific translation task, only the model parameter corresponding to element 1 in the mask matrix corresponding to the translation task is called to participate in the translation result prediction, which reduces the complexity of the model calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a multilingual neural machine translation model training method based on meta-learning and language pair subnet of the present invention;

[0059] Figure 2 It is a meta-path diagram of four English-centered translation tasks;

[0060] Figure 3 It is a flowchart of the meta-learning training process;

[0061] Figure 4 It is a schematic diagram of the multilingual neural machine translation model and the language pair subnetwork structure. DETAILED DESCRIPTION

[0062] Specific implementation method 1: Combination Figure 1 The present embodiment describes a method for training a multilingual neural machine translation model based on meta-learning and language pair subnets, and the method specifically comprises the following steps:

[0063] Step 1: Obtain a multi-language pair dataset, pre-process the multi-language pair dataset, and build a multi-language global shared dictionary based on the pre-processed multi-language pairs; and after adding a language category identifier in front of the pre-processed multi-language pairs, divide the pre-processed multi-language pairs into three parts: training set, validation set, and test set (the division ratio is: each language pair dataset is divided into 80% training set, 10% validation set, and 10% test set, for example, in the French-English dataset, 24,000 items are divided into a training set, 3,000 items are divided into a validation set, and 3,000 items are divided into a test set);

[0064] Step 2: Pre-train the multilingual neural machine translation model based on the meta-learning method, the multilingual global shared dictionary, and the training set, the validation set, and the test set to obtain a pre-trained multilingual neural machine translation model;

[0065] Step 3: Build a subnet for each language pair in the multi-language pair dataset, and the structure of the subnet for each language pair is the same as that of the multi-language neural machine translation model. Each subnet is loaded with the parameters of the pre-trained multi-language neural machine translation model.

[0066] For any language pair in the multi-language pair dataset, fine-tune the parameters of the subnet after loading the parameters using the preprocessing results corresponding to the language pair, and generate a mask matrix for the language pair based on the fine-tuning results;

[0067] Step 4: Use the preprocessed results corresponding to each language pair to train the pre-trained multilingual neural machine translation model to obtain a trained multilingual neural machine translation model;

[0068] When the pre-trained model is trained using the pre-processed result corresponding to the i-th language pair, only the parameter weight corresponding to element 1 in the mask matrix corresponding to the i-th language pair is adjusted;

[0069] Step 5: After preprocessing the sentence to be translated, add a language category identifier to the preprocessing result, and then use the preprocessing result with the language category identifier as the input of the trained multilingual neural machine translation model, and output the translation result through the trained multilingual neural machine translation model.

[0070] Specific implementation method 2: Combination Figure 2 The present embodiment is different from the first embodiment in that the multi-language pair data set is downloaded from IWSLT14, and the multi-language pair data set specifically includes parallel corpora composed of English and French, parallel corpora composed of English and German, parallel corpora composed of English and Spanish, and parallel corpora composed of English and Chinese.

[0071] The other steps and parameters are the same as those in the first embodiment.

[0072] The parallel corpora downloaded in the present invention may include but are not limited to the above-mentioned parallel corpora.

[0073] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the multi-language data set is pre-processed as follows:

[0074] For Chinese corpus: After using the jieba tool to standardize the punctuation marks of the Chinese corpus, the Chinese corpus is segmented.

[0075] For other corpora: the tool libraries Moses and SubwordNMT are used to tokenize the corpus (equivalent to word segmentation), convert uppercase to lowercase, and perform cleaning operations.

[0076] The other steps and parameters are the same as those in the first or second embodiment.

[0077] Specific implementation method 4: This implementation method is different from one of the specific implementation methods 1 to 3 in that the multilingual global dictionary is constructed based on the preprocessing results, specifically:

[0078] Step S101, using a subword segmentation method (BPE) to segment the word segmentation results of each language respectively, so that the number of words obtained after segmentation in each language reaches a preset vocabulary size (the specific size of the vocabulary can be set according to actual conditions);

[0079] Step S102: Merge the vocabulary tables of various languages ​​into a multi-language global shared dictionary.

[0080] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0081] Specific implementation method five: Combination Figure 3 This embodiment is different from the first to fourth embodiments in that the multilingual neural machine translation model is pre-trained based on the meta-learning method, the multilingual global shared dictionary, the training set, the validation set and the test set to obtain a pre-trained multilingual neural machine translation model; the specific process is:

[0082] Step S201, dividing the training set into a support set and a query set, establishing a total task distribution p(τ) including the probability of each translation task occurring during the pre-training process, and the probability of each translation task occurring during the pre-training process can be manually designed, and by designing a balanced probability, the model can quickly adapt to high-resource English-French, English-Spanish, English-Chinese and low-resource English-German language tasks, and is not affected by the data imbalance between high-resource and low-resource languages, thereby improving the translation effect;

[0083] Step S202, setting the inner layer learning rate to α and the outer layer learning rate to β; randomly initializing the parameter θ0 of the multilingual neural machine translation model;

[0084] Step S203, initializing the number of pre-training rounds n=1;

[0085] Step S204, initializing the number of non-cross-task pre-training times of the current round T=1;

[0086] Step S205: Randomly select the T-th pre-trained translation task τ from the total task distribution p(τ) T (A translation task corresponds to only one type of language pair), from the translation task τ T A batch of language pair data is randomly selected from the corresponding data in the support set, and the preprocessing results corresponding to the selected language pair data (the preprocessing results input into the model are all preprocessing results after adding language category identifiers) are used as the input of the multilingual neural machine translation model. The multilingual neural machine translation model outputs translation results according to the multilingual global shared dictionary, and the cross entropy loss function is calculated based on the translation results output by the multilingual neural machine translation model;

[0087] Calculate the gradient according to the cross entropy loss function, and then update the parameters of the multilingual neural machine translation model according to the gradient to obtain the model parameters after the Tth update;

[0088] Step S206, determining whether T=T′ is satisfied, where T′ is the maximum number of non-cross-task pre-training times set;

[0089] If T=T′ is not satisfied, then set T=T+1 and return to step S205;

[0090] If T=T′ is satisfied, then continue to execute step S207;

[0091] Step S207: Use the preprocessing results of the multi-language pairs in the query set to test the T′ group of model parameters obtained by the nth round of non-cross-task pre-training. The T′ group of model parameters all output translation results according to the multi-language global shared dictionary. Then, according to the translation results output by the T′ group of model parameters, update the model parameters after the n-1th round of pre-training to obtain the model parameters after the nth round of pre-training.

[0092] Step S208: determine whether n=N is satisfied, where N represents the maximum number of pre-training rounds set;

[0093] If n=N is not satisfied, n=n+1 is set and the process returns to step S204;

[0094] If n=N, continue to execute step S209;

[0095] Step S209: Use the data in the validation set to verify the training effect of the model parameters after the Nth round of pre-training. If the verification passes (i.e., the translation effect of the model on the validation set meets the requirements), execute step S210; if the verification fails, return to step S203 and continue pre-training the model.

[0096] Step S210: Use the data in the test set to test the verified model. If the test passes (i.e., the translation effect of the model on the test set meets the requirements), a pre-trained multilingual neural machine translation model is obtained. If the test fails, return to step S203 to continue pre-training the model.

[0097] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0098] The present invention optimizes the model across tasks so that the model can quickly adapt to new language pairs with a small amount of data and gradient updates.

[0099] Specific implementation method 6: This implementation method is different from any one of specific implementation methods 1 to 5 in that, in step S205, the parameters of the multilingual neural machine translation model are updated according to the gradient; specifically:

[0100]

[0101] Among them, θ n-1 is the model parameter after the n-1th round of pre-training (for the first round of pre-training, when n=1, θ n-1 =θ0, for the second and subsequent rounds of pre-training, θ n-1 is the model parameter obtained in the previous round of pre-training), θ T ' is the model parameter after the Tth non-cross-task pre-training update, is the cross entropy loss function, yes gradient.

[0102] The other steps and parameters are the same as those in Specific Implementations 1 to 5.

[0103] It should be noted that in the same round of pre-training, each non-cross-task pre-training is to update the network parameters obtained in the previous round of pre-training.

[0104] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that, in step S207, the model parameters after the n-1th round of pre-training are updated according to the translation results output by the T′ group of model parameters to obtain the model parameters after the nth round of pre-training; specifically:

[0105]

[0106] Among them, θ n are the model parameters after the nth round of pre-training, is the loss of the model parameters of the multi-language pairs in the query set after the T-th non-cross-task pre-training update, express gradient.

[0107] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0108] T′ groups of model parameters are model parameters obtained by pre-training based on various types of language pairs. For any type of language pair, the data of the language pair is obtained from the query set, and then the obtained data and the groups of model parameters obtained based on the language pair are used to obtain groups of translation results for the obtained data, and the losses are calculated according to the obtained translation results.

[0109] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the multilingual neural machine translation model adopts a Multilingual Transformer model.

[0110] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0111] The basic architecture of the Multilingual Transformer model is similar to the standard Transformer model. In the encoder of the Multilingual Transformer, a language tag is added before the sentence in the source language to tell the model which language the sentence belongs to. In the decoder, a language tag is also added to the sentence in the target language to help the model distinguish the target language of the translation. The encoder of the Multilingual Transformer model is used to encode the input sequence into a hidden representation, and the decoder is used to decode the hidden representation into a target sequence. The encoder is composed of multiple identical encoder layers stacked together. Each encoder layer contains two sublayers: a multi-head self-attention mechanism and a fully connected feedforward neural network. The decoder is also composed of multiple identical decoder layers stacked together. Each decoder layer has one more multi-head masked self-attention sublayer than the encoder layer. Residual connections and layer regularization operations are applied after each sublayer. Residual connections help alleviate the gradient vanishing and gradient exploding problems, while layer regularization helps speed up model convergence.

[0112] Specific implementation method nine: Combination Figure 4This embodiment is different from any one of the specific embodiments 1 to 8 in that, for any language pair in the multi-language pair data set, the preprocessing result corresponding to the language pair is used to fine-tune the parameters of the subnet after the parameters are loaded, and the mask matrix of the language pair is generated according to the fine-tuning result; the specific process is:

[0113] Step S301: record the data set consisting of the language pair as Among them, D sj→tj represents the jth language pair in the language pair dataset, N represents the number of language pairs in the language pair dataset, sj represents the source language in the language pair, and tj represents the target language in the language pair;

[0114] Step S302, using the language to fine-tune the parameters of the subnet after the parameters are loaded, based on the preprocessing result corresponding to the data set;

[0115] Step S303: prune the model parameters according to the size of the fine-tuned model parameter weights. The layers for parameter pruning specifically include the source language word embedding layer, the multi-head attention layer in the encoder, the feedforward neural network layer, the multi-head attention layer in the decoder, and the feedforward neural network layer. The pruning process is specifically as follows:

[0116] For any layer in the subnet, after sorting all the parameters in the layer in descending order according to the parameter weight, select the parameters with the top α% weight for the layer (the pruning ratio for high-resource languages ​​(English-French, English-Spanish, English-Chinese) is 70%, and the pruning ratio for low-resource languages ​​English-German is 80%), that is, 1-α% of the parameters are pruned, and in the mask vector corresponding to the layer, the values ​​corresponding to the selected parameters are recorded as 1, and the values ​​corresponding to the pruned parameters are recorded as 0;

[0117] Similarly, the mask vectors corresponding to each layer in the subnet are obtained respectively; and the mask matrix of the subnet is composed using the mask vectors corresponding to each layer.

[0118] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0119] The subnet for each language pair is optimized for the specific task of that language pair, so it can better handle the translation task of that language pair without being interfered by other language pairs. The generated mask matrix is ​​actually a matrix of the same size as the model parameters, and each element in the mask matrix indicates whether each parameter is retained. The mask matrix is ​​associated with the subnet corresponding to each language pair. The mask matrix of the subnet retains some shared parameters and maintains specific weights for each language pair, thereby reducing interference while retaining the generalization ability of the multilingual model. In subsequent training, only the weights of the parameters corresponding to element 1 in the mask matrix are updated, and the weights of the parameters corresponding to element 0 in the mask matrix are not involved in the training. This process is called structural perception.

[0120] The complete network and language-specific subnets are Figure 4 As shown in the figure, parameter interference between different language pairs is avoided. The final multilingual translation model is obtained through fine-tuning, pruning and structure-aware training, which further improves the quality of multilingual translation and alleviates the problem of poor performance of low-resource languages ​​in multilingual translation.

[0121] Specific implementation method 10: This implementation method is different from the specific implementation methods 1 to 9 in that the specific process of step S302 is as follows:

[0122]

[0123] in, is the loss function, <x,y> is the Dth sj→tj A sentence pair consisting of a source language corresponding sentence and a target language corresponding sentence in a language pair. Given a source language sentence x, the model has parameters The probability of generating the target language sentence y under

[0124] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0125] The base of the logarithm in the present invention may be 10, but is not limited to 10. This fine-tuning method will increase the weight of important parameters for the type of language, while reducing the weight of unimportant parameters.

[0126] After obtaining the masks for each language pair, the pre-trained model obtained based on the ML method is retrained, and the model θ is trained using batching and structure-aware updates grouped by language pairs. The batching grouped by language pairs creates random batches, and each batch contains only parallel sentences from a single language pair. For example, batch B sj→tj Contains only sentence pairs D from the language pair sj→tj sj→tj , for each language pair, batch B sj→tj Calculate the loss function, the loss function is as follows:

[0127]

[0128] Among them, P θ (y|x) is the probability that the model generates the target language sentence y under the parameters θ given the source language sentence x. The goal is to minimize this loss, that is, to maximize the probability of generating the correct target sentence. During the backpropagation process, only the model parameters related to the current task are updated (that is, the parameters corresponding to element 1 in the mask matrix of the current input language pair). Specifically, only the mask matrix M sj→tj The indicated parameters will be updated to ensure that the specific parameters important for each language pair are optimized. The model parameters are updated through multiple iterations until the training converges. In this way, the specific parameters important for each language pair are optimized based on the training data of the language pair. After training, a unified model θ (including shared parameters and specialized parameters) is finally obtained, which can handle all kinds of language pairs and complete multilingual translation tasks.

[0129] Through the above-mentioned fine-tuning, pruning, and structure-aware training, a model suitable for a variety of specific language pairs can be obtained. Specifically, using θ * Represents the final model parameters after training, which are used in the inference phase to generate prediction results. In the prediction phase, the mask matrix M sj→tj It is applied to the final model parameters, and according to the mask matrix, the model parameters are selected for a specific source language to target language pair, and the selected model parameters are used in the inference process.

[0130] like Figure 4 As shown, during inference, the final model parameter θ * Will be masked M sj→tj Filtering, so that the model only uses the network parameters corresponding to a specific language pair to generate translation results.

[0131] By introducing a specific mask matrix for each language pair, the parameter interference problem in the training of multi-language translation models is solved, and the translation quality of multi-language pairs is improved. At the same time, through fine-tuning, pruning and structure-aware training, each language pair can obtain a dedicated parameter subset. In this way, the model can not only handle multiple language pairs, but also effectively improve the translation performance of each language pair.

[0132] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A multilingual neural machine translation model training method based on meta-learning and language pair subnet, characterized in that: The method specifically comprises the following steps: Step 1: obtain a multi-language pair dataset, pre-process the multi-language pair dataset, and build a multi-language global shared dictionary based on the pre-processed multi-language pairs; add a language category identifier in front of the pre-processed multi-language pairs, and then divide the pre-processed multi-language pairs into three parts: a training set, a validation set, and a test set; Step 2: Pre-train the multilingual neural machine translation model based on the meta-learning method, the multilingual global shared dictionary, and the training set, the validation set, and the test set to obtain a pre-trained multilingual neural machine translation model; Step 3: Build a subnet for each language pair in the multi-language pair dataset, and the structure of the subnet for each language pair is the same as that of the multi-language neural machine translation model. Each subnet is loaded with the parameters of the pre-trained multi-language neural machine translation model. For any language pair in the multi-language pair dataset, fine-tune the parameters of the subnet after loading the parameters using the preprocessing results corresponding to the language pair, and generate a mask matrix for the language pair based on the fine-tuning results; Step 4: Use the preprocessed results corresponding to each language pair to train the pre-trained multilingual neural machine translation model to obtain a trained multilingual neural machine translation model; When the pre-trained model is trained using the pre-processed result corresponding to the i-th language pair, only the parameter weight corresponding to element 1 in the mask matrix corresponding to the i-th language pair is adjusted; Step 5: After preprocessing the sentence to be translated, add a language category identifier to the preprocessing result, and then use the preprocessing result with the language category identifier as the input of the trained multilingual neural machine translation model, and output the translation result through the trained multilingual neural machine translation model.

2. According to claim 1, a multilingual neural machine translation model training method based on meta-learning and language pair subnets is characterized in that: The multi-language pair dataset is downloaded from IWSLT14, and the multi-language pair dataset specifically includes a parallel corpus consisting of English and French, a parallel corpus consisting of English and German, a parallel corpus consisting of English and Spanish, and a parallel corpus consisting of English and Chinese.

3. According to claim 2, a multilingual neural machine translation model training method based on meta-learning and language pair subnets is characterized in that: The multilingual data set is preprocessed as follows: For Chinese corpus: After using the jieba tool to standardize the punctuation marks of the Chinese corpus, the Chinese corpus is segmented. For other corpora: the tool libraries Moses and SubwordNMT are used to tokenize, convert uppercase to lowercase, and clean the corpus.

4. According to claim 3, a multilingual neural machine translation model training method based on meta-learning and language pair subnets is characterized in that: The multilingual global dictionary is constructed according to the preprocessing results, specifically: Step S101, using a subword segmentation method to segment the word segmentation results of each language respectively, so that the number of words obtained after segmentation in each language reaches a preset vocabulary size; Step S102: Merge the vocabulary tables of various languages ​​into a multi-language global shared dictionary.

5. According to claim 4, a multilingual neural machine translation model training method based on meta-learning and language pair subnets is characterized in that: The multilingual neural machine translation model is pre-trained based on the meta-learning method, the multilingual global shared dictionary, and the training set, the validation set, and the test set to obtain a pre-trained multilingual neural machine translation model; the specific process is: Step S201, dividing the training set into a support set and a query set, and establishing a total task distribution p(τ) including the probability of each translation task occurring during the pre-training process; Step S202, setting the inner layer learning rate to α and the outer layer learning rate to β; randomly initializing the parameter θ0 of the multilingual neural machine translation model; Step S203, initializing the number of pre-training rounds n=1; Step S204, initializing the number of non-cross-task pre-training times of the current round T=1; Step S205: Randomly select the T-th pre-trained translation task τ from the total task distribution p(τ) T , from the translation task τ T A batch of language pair data is randomly selected from the corresponding data in the support set, and the preprocessing results corresponding to the selected language pair data are used as the input of the multilingual neural machine translation model. The multilingual neural machine translation model outputs the translation results according to the multilingual global shared dictionary, and the cross entropy loss function is calculated according to the translation results output by the multilingual neural machine translation model; Calculate the gradient according to the cross entropy loss function, and then update the parameters of the multilingual neural machine translation model according to the gradient to obtain the model parameters after the Tth update; Step S206, determining whether T=T′ is satisfied, where T′ is the maximum number of non-cross-task pre-training times set; If T=T′ is not satisfied, then set T=T+1 and return to step S205; If T=T′ is satisfied, then continue to execute step S207; Step S207: Use the preprocessing results of the multi-language pairs in the query set to test the T′ group of model parameters obtained by the nth round of non-cross-task pre-training. The T′ group of model parameters all output translation results according to the multi-language global shared dictionary. Then, according to the translation results output by the T′ group of model parameters, update the model parameters after the n-1th round of pre-training to obtain the model parameters after the nth round of pre-training. Step S208: determine whether n=N is satisfied, where N represents the maximum number of pre-training rounds set; If n=N is not satisfied, n=n+1 is set and the process returns to step S204; If n=N, continue to execute step S209; Step S209: Use the data in the validation set to verify the training effect of the model parameters after the Nth round of pre-training. If the verification passes, execute step S210; if the verification fails, return to step S203 to continue pre-training the model. Step S210: Use the data in the test set to test the verified model. If the test passes, a pre-trained multilingual neural machine translation model is obtained. If the test fails, return to step S203 to continue pre-training the model.

6. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnet according to claim 5, characterized in that: In step S205, the parameters of the multilingual neural machine translation model are updated according to the gradient; specifically: Among them, θ n-1 is the model parameter after the n-1th round of pre-training, θ T ' is the model parameter after the Tth non-cross-task pre-training update, is the cross entropy loss function, yes gradient.

7. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnet according to claim 6, characterized in that: In step S207, the model parameters after the n-1th round of pre-training are updated according to the translation results output by the T′ group of model parameters to obtain the model parameters after the nth round of pre-training; specifically, Among them, θ n are the model parameters after the nth round of pre-training, is the loss of the model parameters of the multi-language pairs in the query set after the T-th non-cross-task pre-training update, express gradient.

8. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnet according to claim 7, characterized in that: The multilingual neural machine translation model adopts the Multilingual Transformer model.

9. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnets according to claim 8, characterized in that: For any language pair in the multi-language pair dataset, fine-tune the parameters of the subnet after the parameters are loaded using the preprocessing result corresponding to the language pair, and generate a mask matrix for the language pair according to the fine-tuning result; The specific process is: Step S301: record the data set consisting of the language pair as Among them, D sj→tj represents the jth language pair in the language pair dataset, N represents the number of language pairs in the language pair dataset, sj represents the source language in the language pair, and tj represents the target language in the language pair; Step S302, using the language to fine-tune the parameters of the subnet after the parameters are loaded, based on the preprocessing result corresponding to the data set; Step S303: prune the model parameters according to the size of the fine-tuned model parameter weights. The layers for parameter pruning specifically include the source language word embedding layer, the multi-head attention layer in the encoder, the feedforward neural network layer, the multi-head attention layer in the decoder, and the feedforward neural network layer. The pruning process is specifically as follows: For any layer in the subnet, after sorting all the parameters in the layer in descending order according to the parameter weight, select the parameters with the top α% weight for the layer, that is, prune the parameters of 1-α%, and in the mask vector corresponding to the layer, record the value corresponding to the selected parameter as 1, and the value corresponding to the pruned parameter as 0; Similarly, the mask vectors corresponding to each layer in the subnet are obtained respectively; and the mask matrix of the subnet is composed using the mask vectors corresponding to each layer.

10. The method for training a multilingual neural machine translation model based on meta-learning and language pair subnet according to claim 9, characterized in that: The specific process of step S302 is as follows: in, is the loss function, <x,y> is the Dth sj→tj A sentence pair consisting of a source language corresponding sentence and a target language corresponding sentence in a language pair. Given a source language sentence x, the model has parameters The probability of generating the target language sentence y under

Images