A Mongolian-Chinese non-autoregressive machine translation method based on multi-task learning

Abstract

A Mongolian-Chinese non-autoregressive machine translation method based on multi-task learning includes preprocessing Mongolian-Chinese parallel corpora; dividing the preprocessed Mongolian-Chinese parallel corpus dataset into a training set, a validation set, and a test set; building an autoregressive translation model with a shared encoder, and forming a multi-task learning framework consisting of the shared encoder, the autoregressive translation model decoder, and the non-autoregressive translation model decoder; within the multi-task learning framework, training the non-autoregressive translation model based on the training set, thereby transferring knowledge from the autoregressive translation model to the non-autoregressive translation model. The obtained non-autoregressive translation model can then be used to perform Mongolian-Chinese translation. This method improves the quality of Mongolian-Chinese translation while ensuring an increased translation rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation, and in particular relates to a Mongolian-Chinese non-autoregressive machine translation method based on multi-task learning. Background Art

[0002] In recent years, machine translation has seen significant growth, driven by the need for communication across different regions. Machine translation for low-resource languages ​​like Mongolian has also gained increasing attention. However, due to a lack of parallel corpora and difficulties extracting semantic features, the translation process between Mongolian and Chinese still suffers from numerous shortcomings, including lengthy training times, inaccurate translations, insufficient semantic information, and inaccurate word vector representations.

[0003] Neural machine translation (NMT), a state-of-the-art machine translation method, has recently adopted two distinct sequence decoding strategies. The first is the autoregressive translation (AT) model, which generates output sequences one by one from left to right, but suffers from slow decoding speed. The second is the non-autoregressive translation (NAT) model, which uses a parallel decoding algorithm to generate output sequences simultaneously, but its translation quality is often lower than that of the autoregressive translation model. Although AT and NAT encoders perform the same sequential learning task, they capture different linguistic properties of the source sentence.

[0004] At present, a series of methods have been proposed to improve non-autoregressive machine translation, introducing technologies such as latent variables and iterative translation knowledge distillation, but these methods have not achieved good results in Mongolian-Chinese translation. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a Mongolian-Chinese non-autoregressive machine translation method based on multi-task learning. Multi-task learning is adopted to transfer the knowledge of the autoregressive translation model to the non-autoregressive translation model through encoder sharing, and the autoregressive translation model is used as an auxiliary task to improve the performance of the non-autoregressive translation model; thereby, it is hoped that the Mongolian-Chinese translation quality can be improved while ensuring the improvement of the translation rate.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A Mongolian-Chinese non-autoregressive machine translation method based on multi-task learning includes the following steps:

[0008] Step 1: Preprocess the Mongolian-Chinese parallel corpus;

[0009] Step 2: Divide the preprocessed Mongolian-Chinese parallel corpus dataset into three parts: training set, validation set and test set;

[0010] Step 3: Build an autoregressive translation model and a non-autoregressive translation model with a shared encoder, and form a multi-task learning framework with the shared encoder, autoregressive translation model decoder, and non-autoregressive translation model decoder.

[0011] Step 4: training the non-autoregressive translation model based on the training set under the multi-task learning framework, thereby transferring knowledge of the autoregressive translation model to the non-autoregressive translation model;

[0012] Step 5: Use the non-autoregressive translation model obtained in step 4 to perform Mongolian-Chinese translation.

[0013] In one embodiment, in step 1, for Chinese, word segmentation is first performed, and then BPE is used for segmentation; for Mongolian, BPE is directly used for segmentation.

[0014] In one embodiment, in step 3, the input of the autoregressive translation model decoder is the result of the previous decoding step. Each decoding step depends on the result of the previous decoding step. When the EOS mark is decoded, the sequence generation process automatically stops to obtain the final decoded sequence. The non-autoregressive translation model decoder has parallel output.

[0015] In one embodiment, the loss function of the multi-task learning framework at an iteration step size t is defined as the weighted sum of the losses of the autoregressive translation model and the non-autoregressive translation model:

[0016]

[0017] Among them, L at and L nat are the losses of the autoregressive translation model and the non-autoregressive translation model respectively; θ enc 、 are the parameters of the shared encoder, autoregressive translation model decoder, and non-autoregressive translation model decoder respectively; at the iteration step size t, λ t Parameters for adjusting the balance between autoregressive and non-autoregressive translation models:

[0018]

[0019] Where T is the total number of training steps, L at The weights of are linearly annealed from 1.0 to 0.0 during the training process, and L nat The weight of increases from 0.0 to 1.0;

[0020] During model training, Mongolian sentence X is input into the encoder, and Chinese sentence Y is input into the autoregressive translation model decoder and the non-autoregressive translation model decoder respectively. During the training process, only the Chinese translation generated by the non-autoregressive translation model decoder is used, where the Chinese sentence Y is the Chinese translation corresponding to Mongolian sentence X in the dataset.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The encoders of non-autoregressive and autoregressive translation models can capture different properties of the source sentence. Specifically, while previous non-autoregressive translation models significantly improved decoding speed, they also reduced translation quality. Therefore, this paper uses a multi-task learning approach for Mongolian-Chinese translation. By sharing encoders, knowledge from the autoregressive machine translation model is transferred to the non-autoregressive machine translation model, using the autoregressive machine translation model as an auxiliary task to improve the performance of the non-autoregressive model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Distinguish between autoregressive and non-autoregressive translation models.

[0024] Figure 2 A multi-task learning framework. DETAILED DESCRIPTION

[0025] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0026] The present invention is a Mongolian-Chinese non-autoregressive machine translation method based on multi-task learning, comprising the following steps:

[0027] Step 1: Preprocess the Mongolian-Chinese parallel corpus.

[0028] Because Chinese, unlike Mongolian and English, does not have spaces between words, it first undergoes word segmentation. This example uses the popular Jieba Chinese word segmentation technology, followed by BPE segmentation. For Mongolian, BPE segmentation is used directly. This word segmentation process can alleviate the impact of low-frequency words to a certain extent.

[0029] Step 2: Divide the preprocessed Mongolian-Chinese parallel corpus dataset into three parts: training set, validation set and test set.

[0030] The training set is used to train the translation model, and its quality determines the model's performance. The validation set allows the model to monitor translation performance in real time during training. The test set is used to test the final model's translation performance after training.

[0031] The ratio of the three parts can be set manually based on conventions or other reference factors. For example, the ratio of the training set, validation set, and test set can be 8:1:1.

[0032] Step 3: Build an autoregressive translation model and a non-autoregressive translation model with a shared encoder, and form a multi-task learning framework with the shared encoder, autoregressive translation model decoder, and non-autoregressive translation model decoder.

[0033] The autoregressive translation model and the non-autoregressive translation model of the present invention use the same encoder. Figure 1 , <bos>Indicates the start of a sentence. The input of the autoregressive translation model decoder is the result of the previous decoding step. Each step of decoding generation depends on the result of the previous decoding step. When the sentence end symbol is decoded <eos>When , the sequence generation process automatically stops, and the final decoded sequence is obtained. The non-autoregressive translation model does not have such a dependency characteristic, which enables parallel output of the decoder and improves the translation speed.

[0034] Step 4: Under the multi-task learning framework, a non-autoregressive translation model is trained based on the training set, thereby transferring the knowledge of the autoregressive translation model to the non-autoregressive translation model.

[0035] The present invention uses an additional autoregressive translation task as an auxiliary task. The encoder parameters of this task are shared with the non-autoregressive translation task, while the decoder parameters are mutually exclusive. In view of the autoregressive translation model and the non-autoregressive translation model under the standard encoder structure, the present invention will adopt a hard parameter sharing method to share their encoder parameters. Figure 2 The figure shows the multi-task learning framework adopted by the present invention. The framework consists of three parts: a shared encoder, an autoregressive translation model decoder, and a non-autoregressive translation model decoder.

[0036] When translating the target sentence, the autoregressive translation model translates word by word from left to right. Specifically, given a Mongolian sentence For possible output Chinese sentences Generate it as follows:

[0037]

[0038] Here, t represents the current iteration step, Y ＜t Indicates the target word that has been generated, θ enc 、 are the parameters of the shared encoder and the decoder of the autoregressive translation model, respectively.

[0039] is the last word of the output Chinese sentence. Y Y is the total number of words in the output Chinese sentence. t is the tth output Chinese word. Is the last word of a Mongolian sentence. X is the total number of words in a Mongolian sentence.

[0040] Performing an autoregressive decomposition on the output distribution allows direct maximum likelihood training and applying the cross-entropy loss of the autoregressive translation model at iteration step t:

[0041]

[0042] When the non-autoregressive translation model translates the target sentence, after removing the dependencies between words when generating the target sentence, the generation process can be expressed as follows:

[0043]

[0044] z is the input to the decoder of the non-autoregressive translation model.

[0045] The encoder of the autoregressive translation model is exactly the same as that of the non-autoregressive translation model. In the decoder during training, the input of the autoregressive translation model is the translation corresponding to the Mongolian sentence X, and the input of the non-autoregressive translation model is z in the above formula:

[0046] z = f(x; θ enc )

[0047] f() represents the operation performed by the productivity prediction module.

[0048] z is obtained by copying the source sentence X. For example, when translating the sentence "Hello, world.” into Chinese, if the output translation is "你好,世界。” and Hello is translated into Chinese with two characters, then the productivity value is 2; world is also translated into Chinese with two characters, so the productivity value is 2. So here X is "Hello, world.”, and z can be simply understood as "HelloHello, world world”.

[0049] The parallel output of the non-autoregressive translation model is achieved by copying each input word of the encoder zero or more times as an input to the decoder. The number of times each input is copied is called the "productivity” of the input word. The productivity value of each word is determined by the length of the word after being translated into the target translation, and the resulting output length is determined by the sum of the productivity values of all words. A productivity prediction module is introduced to predict how many times each word in X will be copied during the copying process. The maximum likelihood loss function, which is the loss of the non-autoregressive translation model, is as follows:

[0050]

[0051] In the multi-task learning framework used in this invention, the encoder parameters of the autoregressive and non-autoregressive translation models are shared. The overall loss function of the framework at iteration step t is defined as the weighted sum of the losses of the autoregressive translation model and the non-autoregressive translation model:

[0052]

[0053] where, L at and L nat are the losses of the autoregressive translation model and the non-autoregressive translation model respectively; θ enc 、 are the parameters of the shared encoder, the decoder of the autoregressive translation model, and the decoder of the non-autoregressive translation model respectively; at iteration step t, λ t Parameters for adjusting the balance between autoregressive and non-autoregressive translation models:

[0054]

[0055] Where T is the total number of training steps; since L at It only plays an auxiliary role and does not directly affect the non-autoregressive translation model, so it is directly used to reduce the importance of AT loss near the end of the training process. Under this scheme, L at The weights of are linearly annealed from 1.0 to 0.0 during the training process, and L nat The weight of increases from 0.0 to 1.0.

[0056] During model training, Mongolian sentence X is input into the encoder, and Chinese sentence Y is input into the autoregressive translation model decoder and non-autoregressive translation model decoder respectively. During the training process, only the Chinese translation generated by the non-autoregressive translation model decoder is used, where Chinese sentence Y is the Chinese translation corresponding to Mongolian sentence X in the dataset.

[0057] Step 5: Use the non-autoregressive translation model obtained in step 4 to perform Mongolian-Chinese translation.

[0058] In one embodiment of the present invention, For example, the source language sentence is divided into Correspondingly, the translation "Today is sunny" is split into "-, today, is, sunny". The data is fed into the autoregressive and non-autoregressive translation models, respectively. The autoregressive model generates the translation "Today is sunny," while the non-autoregressive model generates the translation "Today is sunny" in parallel. During the translation process, the encoder parameters of the two models are shared, and the final output is the translation "Today is sunny" generated by the non-autoregressive model.< / eos> < / bos>

Claims

1. A Mongolian-Chinese non-autoregressive machine translation method based on multi-task learning, characterized in that: The steps include: Step 1: Preprocess the Mongolian-Chinese parallel corpus; Step 2: Divide the preprocessed Mongolian-Chinese parallel corpus dataset into three parts: training set, validation set and test set; Step 3: Build an autoregressive translation model and a non-autoregressive translation model with a shared encoder. A multi-task learning framework is formed by the shared encoder, the autoregressive translation model decoder, and the non-autoregressive translation model decoder. The input of the autoregressive translation model decoder is the result of the previous decoding step. Each decoding step depends on the result of the previous decoding step. When the EOS mark is decoded, the sequence generation process automatically stops, and the final decoded sequence is obtained. The decoder of the non-autoregressive translation model has parallel outputs; The loss function of the multi-task learning framework at iteration step t is defined as the weighted sum of the losses of the autoregressive translation model and the non-autoregressive translation model: Among them, L at and L nat are the losses of the autoregressive translation model and the non-autoregressive translation model respectively; θ enc 、 are the parameters of the shared encoder, autoregressive translation model decoder, and non-autoregressive translation model decoder respectively; at the iteration step size t, λ t Parameters for adjusting the balance between autoregressive and non-autoregressive translation models: Where T is the total number of training steps; L at The weights of are linearly annealed from 1.0 to 0.0 during the training process, and L nat The weight of increases from 0.0 to 1.0; During model training, Mongolian sentence X is input into the encoder, and Chinese sentence Y is input into the autoregressive translation model decoder and the non-autoregressive translation model decoder respectively. During the training process, only the Chinese translation generated by the non-autoregressive translation model decoder is used, where the Chinese sentence Y is the Chinese translation corresponding to Mongolian sentence X in the dataset; The encoders of the autoregressive translation model and the non-autoregressive translation model are exactly the same. In the decoder, during training, the input of the autoregressive translation model is the translation corresponding to the Mongolian sentence X, and the input of the non-autoregressive translation model is the input z of the decoder of the non-autoregressive translation model, z = f(x; θ enc ), f() represents the operation performed by the generating force prediction module; The parallel output of the non-autoregressive translation model is achieved by replicating each input word of the encoder zero or more times as an input to the decoder. The number of times each input is replicated is called the generative power of the input word. The generative power value of each word is determined by the length of its corresponding word after translation into the target text, and the resulting output length is determined by the sum of the generative power values ​​of all words. The generative power prediction module predicts how many times each word in the Mongolian sentence X will be replicated during the replication process. Step 4: training the non-autoregressive translation model based on the training set under the multi-task learning framework, thereby transferring knowledge of the autoregressive translation model to the non-autoregressive translation model; Step 5: Use the non-autoregressive translation model obtained in step 4 to perform Mongolian-Chinese translation.

2. The Mongolian-Chinese non-autoregressive machine translation method based on multi-task learning according to claim 1 is characterized in that: In step 1, for Chinese, word segmentation is first performed, and then BPE is used for segmentation; for Mongolian, BPE is directly used for segmentation.

Images