Small sample text data hybrid enhancement method

Abstract

The small sample text data hybrid enhancement method disclosed by the invention is simple, complete and high in self-adaption. The method is realized through the following technical scheme: based on a text data enhancement target, dividing an original text into long text data and short text data, automatically separating and distinguishing the long text data and the short text data, carrying out synonym replacement, random insertion, random exchange and random deletion on the long text data, automatically adapting texts with different lengths, carrying out retranslation enhancement on the short text data, carrying out statistical analysis on text data sample length distribution, subdividing data sample distribution into groups with finer granularity, and carrying out mask prediction or pre-training; classifying each text data sample into different groups, setting different covering probabilities for the text data samples of different groups according to the groups, and performing mask prediction through a noise reduction self-encoding process to realize secondary enhancement of the text data; and generating batch enhanced texts according to the small sample quantity to realize small sample text data hybrid enhancement. The text enhancement quantity is improved, and the enhancement quality is ensured.

Description

technical field [0001] The present invention relates to many information processing fields such as artificial intelligence and natural language processing, and is mainly used for data enhancement technology of text classification, especially related to comprehensive text data enhancement technology. Background technique [0002] Data augmentation, the artificial creation of training data by transformation for machine learning, is an area of ​​study that is widely studied across machine learning disciplines. It is not only useful for improving the generalization ability of the model, but also can solve many other challenges and problems, from overcoming the limited amount of training data to regulating the goal to limiting the amount of data used to preserve privacy. Data augmentation refers to the data augmentation or feature enhancement of the original small sample data set with the help of auxiliary data or auxiliary information. Data augmentation is to add new data to the...

AI Technical Summary

Problems solved by technology

[0005] In order to further improve the quantity and quality of small-sample text data, the present invention aims at the problem of incomplete text enhancement methods in existing applications, and provides a simple, complete, self-adaptive, relatively stable and effective small-sample text data mixed enhancement method , which is beneficial to downstream tasks such as subsequent text classification

Images