Encoder-decoder framework pre-training method for neural machine translation

Abstract

The invention discloses an encoder-decoder framework pre-training method for neural machine translation. The encoder-decoder framework pre-training method comprises the steps of: constructing a largenumber of multi-language document-level monolingual corpora, and adding a special identifier in front of each sentence to represent the language type of the sentence; processing sentence pairs to obtain training data; training monolingual data of different languages to obtain converged pre-training model parameters; constructing parallel corpora, and initializing parameters of a neural machine translation model by using the pre-training model parameters; finely adjusting model parameters of the initialized neural machine translation model through parallel corpora to finish a training process;and in a decoding stage, encoding a source language sentence by using an encoder of the trained neural machine translation model, and decoding by using a decoder to generate a target language sentence. According to the encoder-decoder framework pre-training method, the model has language modeling capability and language generation capability, the pre-training model is applied to the neural machinetranslation model, the convergence rate of the model can be increased, and the robustness of the model is improved.

Description

technical field [0001] The invention relates to a pre-training method for an encoder-decoder framework, in particular to a neural machine translation-oriented encoder-decoder framework pre-training method. Background technique [0002] In the neural network, the pre-training method refers to obtaining a basic model through massive general data training. This general and sufficient data can encourage the model to have good generalization ability on downstream tasks in the same field. Afterwards, for downstream tasks, the pre-trained model is fine-tuned using task-specific data, so that the model pays more attention to task-related features and has better performance on this task. In the case of small amount of task-specific data, the pre-training method can effectively improve the performance of the model, and since the pre-training model already has general feature extraction capabilities, the fine-tuning model can achieve faster convergence and stronger robustness sex. ...

AI Technical Summary

Problems solved by technology

However, this integration method has an important shortcoming, that is, only part of the information of the pre-trained model can be applied to the neural machine translation model, or it can only be applied to some modules of the neural machine translation model, and the entire model cannot be initialized. Some parameters need to be learned from scratch, which limits the benefits of pre-trained models

Images