Corpus enhancement method and device, computer equipment and readable storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, and provides a corpus enhancement method and device, computer equipment and a readable storage medium, the corpus enhancement method comprises the following steps: obtaining a dictionary and a corpus, according to a first word frequency and a second word frequency, vectorizing a corpus without a label in the corpus, and obtaining an N-dimensional feature matrix; obtaining labels in the corpus, performing matrix processing on the N-dimensional feature matrix according to the labels, and obtaining a first matrix used for bearing semantics of the labels and words and a second matrix used for bearing semantics of the labels and corpora; the similarity between the first matrix and the second matrix is obtained, and the labels are distributed to the linguistic data without the labels according to the similarity. The enhanced corpus can map the association relationship among the tag, the corpus, the dictionary and the untagged corpus, so that the confidence coefficient of the tag allocated to the untagged corpus is improved, and the purpose of providing high-quality corpus for model training is achieved.

Description

technical field [0001] The present invention is designed in the technical field of artificial intelligence, and in particular, relates to a corpus enhancement method, apparatus, computer equipment and readable storage medium. Background technique [0002] In recent years, with the development of artificial intelligence (Artificial Intelligence, AI) technology, the technological progress of natural language processing (NLP) has been promoted, and the precision of natural language processing is getting higher and higher. In the training of natural language processing models, a large amount of model data is required, especially in some processing tasks, there are often insufficient data volume of model data, great difficulty in labeling, and inaccurate labeling. Therefore, it is necessary to perform data enhancement processing on the marked corpus to expand more high-quality corpora. [0003] At present, corpus enhancement methods include synonym replacement, entity replacemen...

AI Technical Summary

Problems solved by technology

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a corpus enhancement method, device, computer equipment and readable storage medium for solving the problem of low quality of the enhanced corpus in the prior art

Images