An Active Learning Short Text Classification Method and System Based on Sampling Frequency Optimization

Abstract

The invention discloses an active learning short text classification method and system based on sampling frequency optimization, broadens the active learning optimization direction, and provides a simple and effective optimization framework widely used in the industry. The technical scheme is as follows: the text classifier learns the marked data; the unmarked data is sampled and evaluated based on the learning result of the text classifier and the most valuable data is selected; the selected data is manually marked and added to the marked data, Repeat the above steps until the number of iterations reaches the upper limit or the accuracy reaches the target. In the sampling evaluation process, the labeled data is classified according to its category, the amount of labeled data in each category is counted, and the respective sampling frequency data is obtained; for the unlabeled data, the unlabeled data is evaluated first to obtain the initial evaluation. score and its prediction result category, and then obtain the corresponding sampling frequency data according to the prediction result category, and obtain the final evaluation score based on the initial evaluation score and the sampling frequency data of the corresponding category.

Description

technical field [0001] The invention relates to a technique for using active learning to process short text classification, in particular to an active learning short text classification method and system based on sampling frequency optimization. Background technique [0002] With the prevalence of e-commerce and online communication, in many application fields, such as instant messaging, online chat logs, bulletin board system headlines, Internet news comments, Twitter, etc., short text content is flooding people's daily life. In the face of many needs such as topic recommendations and e-commerce chat robots derived from this, short text classification becomes very important. However, due to the non-standard expression of short text, labeling the required content will consume a lot of manpower, and the effect is relatively limited. [0003] At present, the industry uses active learning to deal with short text classification problems, but when the mainstream active learning ...

AI Technical Summary

Problems solved by technology

However, due to the non-standard expression of short text, labeling the required content will consume a lot of manpower, and the effect is relatively limited

[0003] At present, the industry uses active learning to deal with short text classification problems, but when the mainstream active learning methods are directly applied to this, we find that the performance in short text classification is not good, and with the increase of the number of categories in the data set, the existing The performance of the sampling method will also decrease accordingly

Images