Training-corpus quality evaluation and selection method orienting to statistical-machine translation

Abstract

The invention relates to a training-corpus quality evaluation and selection method orienting to statistical-machine translation. The training-corpus quality evaluation and selection method comprises the following steps of: automatic weight acquisition: adopting small-scale corpus to train an automatic weight acquisition model so as to obtain a characteristic weight and a classification critical value; sentence-pair quality evaluation: using the weight and the classification critical value as well as the original large-scale parallel corpuses as input, carrying out classification on the large-scale parallel corpuses by using a linear model for sentence-pair quality evaluation, and generating all corpus subsets; and high-quality corpus subset selection: on the basis of all the corpus subsets, considering the influence of the cover degree, and selecting the high-quality corpuses as training data of a statistical-machine translation system. The training-corpus quality evaluation and selection method has the advantages that richer sequence-pair quality evaluation characteristic is provided, so that the automatic learning of the characteristic weight is realized, and when the scale of the subsets reaches to 30%, the performance can reach 100%, even better; and the class of any input sequence pair can be divided, and help can be provided for tasks such as selection of high-quality corpus data.

Description

technical field [0001] The invention relates to a statistical machine translation technology, in particular to a statistical machine translation-oriented training corpus quality evaluation and selection method. Background technique [0002] The training of the Statistical Machine Translation (SMT) system requires the support of large-scale bilingual parallel corpus, and the quality and quantity of the corpus will have a great impact on the performance of the machine translation system. In general, increasing the size of the training corpus helps to obtain stable model parameters and improve the translation performance of SMT systems. However, the larger the corpus, the higher the execution cost of the system, and the longer it takes to train and decode. In addition, a larger corpus means that there may be more noisy data, which will affect the reliability of system training to a certain extent. [0003] Yao Shujie et al. (2010) proposed a method for selecting statistical m...

AI Technical Summary

Problems solved by technology

Manual methods require a lot of experimental support, are time-consuming and labor-intensive, and become very unusable when the number of features increases

Images