A Text Classification Method with Missing Negative Examples

Abstract

The invention discloses a text classification method lacking negative examples, which belongs to the technical field of machine learning and text classification. This method first determines the data text to be classified, and customizes the text classification category; then trains the TF-IDF model and the LSI model based on the acquired corpus; then builds the feature vector of the text based on the trained TF-IDF model and the LSI model, And build a combined text feature vector based on the ensemble method; then use the ROC-SVM combination algorithm to train the Basic classifier, and can combine the k-means clustering method to train the Basic classifier and train the label classifier at the same time; finally, the to-be-classified The text of the text is initially classified by the Basic classifier, and then screened by Elasticsearch to determine the candidate classification, and then the label classifier is used to accurately classify the documents to be classified into one or several categories in the custom category. The method of the invention can effectively classify text data lacking negative examples, and has high accuracy, good effect and high efficiency.

Description

technical field [0001] The invention belongs to the technical field of machine learning and text classification, and in particular relates to a text classification method lacking negative examples. Background technique [0002] With the development of the Internet, the number of Internet texts has increased dramatically, and the resulting demand for textual classification has also become stronger. In the face of massive data texts, manual classification is obviously impossible, but with the rise of machine learning methods, it provides ideas to solve this demand. Therefore, a large number of researchers have proposed a series of methods around this field. For example, machine learning methods such as naive Bayesian method, decision tree, k-nearest neighbors, and support vector machines have been successfully applied to text classification and achieved good results. However, because the data texts in different fields are intricate and the mechanisms of many methods are diffe...

AI Technical Summary

Problems solved by technology

Research in this area, such as the LSI (Latent Semantic Indexing) model, is sometimes called LSA (Latent Semantic Analysis), which can obtain text compression information to a large extent through SVD (Singular Value Decomposition), but such methods also exist Some defects, such as the obtained model cannot be explained by probability, lack of statistical theory support, etc.

[0007] (2) In the absence of negative example data, it is difficult to select training data for classification models

For example, PCA, decision tree, Bayesian framework, S-EM and other methods are used by scholars in the selection process of negative examples, but these are not classifiers with strong generalization ability, so the final effect is slightly lacking

[0008] (3) When there are many categories that need to be judged, it will take a lot of time to use the trained classification model to score each category, which will seriously affect the use in the production environment

[0009] Through the above analysis, when performing text classification, for text classification lacking negative example data, the classification is difficult due to the lack of negative example data, and the accuracy rate is poor, the effect is poor, and the efficiency is low

Images