Data enhancement method and system based on multilingual machine translation

Abstract

The invention relates to a data enhancement method and system based on multilingual machine translation. The method comprises the steps: translating an original sentence from a source language to a target language through a pre-trained multilingual translation model to obtain a plurality of candidate translations with different probabilities, the source language being the same as the target language; and reserving candidate translations which are not completely the same as the original sentences in the plurality of candidate translations as training data for training corresponding translation models. Due to joint training and knowledge migration, translation of low-resource and zero-resource languages in the multilingual translation model is benefited from high-resource languages, so that high-quality and diversified training data can be obtained as training resources for training the translation model, and improvement of model performance is facilitated; moreover, since the multilingual translation model has the capabilities of multilingual translation and zero-resource translation, translation among the same languages of multiple languages can be realized, so that data enhancement of the multiple languages can be carried out only by training one multilingual translation model, and the resource cost is reduced.

Description

technical field [0001] The invention relates to the field of translation and artificial intelligence, in particular to a data enhancement method and system based on multilingual machine translation. Background technique [0002] The general-purpose neural machine translation system is based on the end-to-end Encoder-Decoder framework. In general, a large-scale parallel corpus is required for supervised model training. However, due to the high cost of manual labeling, bilingual corpora in low-resource languages , usually use data enhancement methods such as back-translation and data noise to forge parallel corpus and add them to the training data for training, but data noise has limited improvement in data diversity; the quality of data generated by back-translation depends on the translation model , and the effect of the translation model depends on the size of the parallel corpus, so it is not conducive to data generation in low-resource languages, and when the number of la...

AI Technical Summary

Problems solved by technology

[0002] The general-purpose neural machine translation system is based on the end-to-end Encoder-Decoder framework. In general, a large-scale parallel corpus is required for supervised model training. However, due to the high cost of manual labeling, bilingual corpora in low-resource languages , usually use data enhancement methods such as back-translation and data noise to forge parallel corpus and add them to the training data for training, but data noise has limited improvement in data diversity; the quality of data generated by back-translation depends on the translation model , and the effect of the translation model depends on the size of the parallel corpus, so it is not conducive to data generation on low-resource languages, and when the number of languages ​​increases, more translation models need to be trained

Images