Corpus text processing method and device and electronic equipment

Abstract

The invention provides a corpus text processing method and device and electronic equipment. The method comprises the steps of inputting a corpus text set to be processed into a language model, and obtaining feature vectors of corpus texts; performing clustering processing on the corpus text set based on a clustering algorithm and the feature vectors of the corpus texts to obtain corpus classification information; modifying intention category annotation information annotated by the target corpus text to obtain the target corpus text; and adding the target corpus text into the original trainingsample to train a language model to obtain an optimized language model. According to the invention, clustering processing is carried out on the corpus text set through the language model and the clustering algorithm, and the intention category annotation information annotated by the target corpus information in the corpus classification information is corrected to train the language model, so thatthe language model can be iteratively optimized in the use process, the generalization ability of the language model and the clustering algorithm is improved, and the labeling accuracy of the intention category labeling information corresponding to the corpus text is improved.

Description

technical field [0001] The present invention relates to the technical field of natural language processing, in particular to a method, device and electronic equipment for processing corpus texts. Background technique [0002] With the rapid development of computers, the number of digitized texts is constantly increasing, and the development of the Internet has intensified the expansion of digitized texts. In this context, clustering technology can be used to simplify the representation of text and re-express information retrieval to speed up information retrieval; or realize the integration and push of a series of personalized information, such as the currently popular APP (Application , mobile phone software) Toutiao, Zhihu, etc. However, in most scenarios, chatbots still need to customize specific question-answer pairs, that is, the pairing of intentions and answers. This mode is very common in task-based conversations, such as booking airline tickets. But in open domain...

AI Technical Summary

Problems solved by technology

[0003] The existing corpus texts mainly use clustering algorithm and metric learning to mark the information of intent categories. Among them, the model of metric learning is the traditional sequence model, which has limited ability to express corpus texts, which leads to the accuracy of labeling information of intent categories. lower

Images