A classification method and device for unlabeled corpus

Abstract

The present invention provides a method and device for classifying unlabeled corpus, said method comprising: obtaining unlabeled corpus, said unlabeled corpus including at least one question; inputting each question included in said unlabeled corpus into a text classification model , output the label corresponding to each question; wherein, the text classification model is obtained after training based on an unlabeled corpus sample, and each piece of corpus data in the unlabeled corpus sample includes a question and an answer. The device is used to perform the above method. The method and device for classifying unlabeled corpus provided by the embodiments of the present invention improve the accuracy of classifying unlabeled corpus.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence, in particular to a method and device for classifying unlabeled corpus. Background technique [0002] With the development of artificial intelligence technology, artificial intelligence-based dialogue robots have been widely used in many fields such as customer service, outbound calls, sales, intelligent search, etc., and intent recognition, as the core technology in the dialogue robot system, directly determines the accuracy of the dialogue and user experience. [0003] At present, in the intent recognition technology, the more effective technology is the deep learning model. The deep learning model obtained through training can realize the classification of unlabeled corpus, which is helpful for the recognition of intent. However, the training of deep learning models needs to collect and label a large number of samples, which is very time-consuming and labor-intensive, and the a...

AI Technical Summary

Problems solved by technology

However, the training of deep learning models needs to collect and label a large number of samples, which is very time-consuming and labor-intensive, and the accumulation of a large amount of labeled data (ie sample data) takes a very long time, and a large number of high-quality labeled data is very expensive.

Another thing is that there are too many parameters in the deep learning model, and it is easy to "overfit" when the sample data is small, and it is very sensitive to noise data.

In order to solve the overfitting problem caused by less sample data, the existing technology adopts a simple model when selecting a model, and uses technologies such as penalty items, and uses technologies such as denoising and sample expansion in data processing, but it is still difficult to solve the problem of sample data. Too little data leads to the problem that the trained deep learning model is not accurate enough, resulting in low classification accuracy of unlabeled corpus, which affects the application of the deep learning model

Images