Training corpus acquisition method and apparatus

Abstract

The present invention provides a training corpus acquisition method and apparatus, and has the advantages of being high in automation degree and quick in acquisition speed. The method comprises: obtaining a first initial training corpus and a second initial training corpus; performing prediction on an optional training statement by using a probabilistic classification model constructed according to the first initial training corpus, so as to obtain a first prediction result; performing prediction on the optional training statement by using a probabilistic classification model constructed according to the first initial training corpus and the second initial training corpus, so as to obtain a second prediction result; and comparing the first prediction result with the second prediction result; if classification information of the first prediction result is inconsistent with that of the second prediction result, or if the classification information of the first prediction result is consistent with that of the second prediction result and a prediction probability of the first prediction result is less than that of the second prediction result, using the optional training statement and the classification information of the second prediction result as a training corpus and outputting the training corpus.

Description

technical field [0001] The present invention relates to the technical field of natural language processing, in particular to a training corpus acquisition method and device. Background technique [0002] Intent recognition, that is, to recognize the intention of a behavior. For example, in a question-and-answer dialogue, each sentence of the questioner has a certain intention, and the respondent answers according to the intention of the other party. Intent recognition has a wide range of applications in scenarios such as search engines and chat robots. [0003] The existing intent recognition methods mainly obtain a batch of corpus, and manually mark the intent of each corpus to obtain training data. A probabilistic classification model is trained by combining the training data with a specific algorithm, and the resulting probabilistic classification model is used to identify the intent of the new corpus. Due to the small amount of initially manually annotated corpus, som...

AI Technical Summary

Problems solved by technology

All data in this method needs to be manually extracted, and only one problem can be found to solve one. Many wrong problems are missed due to human reasons, a

Images