Text corpus expansion method

Abstract

The invention belongs to the technical field of natural language processing, and particularly provides a text corpus expansion method which is used for overcoming the defects of an existing text enhancement method in sentence pattern diversity and semantic fidelity of corpus generation. According to the method, a word replacement method and a translation method are combined, and the defects causedby combined use of the two methods are overcome; firstly, synonym hash tables, input sentences and the like are subjected to standardization processing, and meanwhile, a strategy of performing one-way replacement in the direction that the number of words is not reduced is adopted in word replacement, so that the influence of words with wrongly written characters and spoken words on the back-translation effect is reduced; in addition, by introducing masking translation, the problem of inaccurate translation of the professional noun by a translation method is improved; the corpus generated by the method gives consideration to sentence pattern diversity and semantic fidelity, and is particularly suitable for text corpus expansion application under the condition that the initial corpus is insufficient.

Description

technical field [0001] The invention belongs to the technical field of natural language processing, and in particular relates to a text corpus expansion method. Background technique [0002] When deep learning technology is applied to natural language processing, it often encounters the problem of insufficient text training data or data imbalance. Solving the problem of lack of text corpus has become the focus of research in the field of natural language processing. Text enhancement is a common method to increase text training corpus, which is to construct more text data through some processing on the basis of existing text data. Unlike image data enhancement, text is discrete, and data cannot be increased by linear interpolation, rotation, etc., and the semantic information and grammatical structure of sentences need to be considered. [0003] Text enhancement methods mainly include manual annotation, word replacement, back translation, neural network and other methods. M...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to provide a text corpus expansion method for the problems of the above-mentioned prior art. The corpus generated by this method can take into account both the diversity of sentence patterns and the fidelity of semantics, and is especially suitable when the initial corpus is not sufficient. Text corpus augmentation application

Images