Method and device for optimizing corpus

Abstract

The invention provides a method for optimizing a corpus and a device for optimizing the corpus. The device for optimizing the corpus comprises a filter unit, a replacement unit and a perplexity calculation unit; the filter unit performs filtration on sentences in the corpus on the basis of an optimization parameter to obtain sentence pairs to be optimized; the replacement unit performs replacement on at least part of the sentence pairs to be optimized; the perplexity calculation unit calculates the perplexity of the replaced sentence pairs, and the replaced sentence pairs can serve as optimization results of the sentence pairs to be optimized on the condition that the perplexity of the replaced sentence pairs is smaller than that of the sentence pairs to be optimized.

Description

technical field [0001] The present invention relates to the technology of natural language processing, in particular, to a method and a device for optimizing a corpus. Background technique [0002] The performance of statistical machine translation depends largely on the quantity and quality of parallel corpora. On the one hand, the collected large-scale training data needs to be efficiently managed for different purposes. On the other hand, the more corpus, the more noise it contains, and these noises will affect the estimation accuracy of translation model and language model. Therefore, filtering noise in training corpus is a very basic and important task. In this regard, the following methods exist in the prior art. [0003] (1) English-Chinese bilingual corpus filtering method, including the following steps: A. Determine the sentence length ratio feature value of the English-Chinese bilingual sentence pair; The quantity of the corresponding word matching in the bilin...

AI Technical Summary

Problems solved by technology

On the other hand, the more corpus, the more noise it contains, which will affect the accuracy of the translation model and language model estimation

Images