Unbalanced text classification method and system combining SVM and semi-supervised clustering

Abstract

The invention discloses an unbalanced text classification method and system combining SVM and semi-supervised clustering. The unbalanced text classification method comprises the steps: carrying out preprocessing on a to-be-processed text, and obtaining text data in a vector format, and enabling the text data to serve as a data set; using the training set to train the SVM classifier to obtain a classification model, and using the classification model to predict the test set to obtain the category and confidence of the test set; clustering the data set by using a semi-supervised clustering algorithm to obtain the category to which the test set belongs and the confidence coefficient of the test set; and fusing the category to which the test set obtained by the SVM classifier and the semi-supervised clustering algorithm belongs and the confidence coefficient of the test set to obtain final output. According to the unbalanced text classification method, different types of methods in the technical field of unbalanced text classification are combined; advantage complementation of the different methods is achieved; vectorization and normalization methods are used; and the defect that whenhigh-dimensional sparse text data are processed, a text classification result is inaccurate due to the fact that labeled texts are too few is overcome. The unbalanced text classification method effectively solves the problem of text class imbalance.

Description

technical field [0001] The invention belongs to the field of natural language processing, in particular to the field of unbalanced text classification, in particular to an unbalanced text classification method and system combining SVM and semi-supervised clustering. Background technique [0002] Text classification is a classic problem in the field of natural language processing, and it is widely used in information filtering, mail classification, query intent prediction, text topic tracking and other fields. Traditional text classification methods are mainly designed for balanced text classification problems, and they work well when dealing with small-scale balanced text classification problems with uniform and dense data distribution. But there are still many limitations. Especially in practical applications, due to the characteristics of category imbalance, too few labeled texts, and high-dimensional sparse samples, the complexity of text classification is increased, res...

AI Technical Summary

Problems solved by technology

[0008] According to the problems existing in the prior art, the present invention proposes an unbalanced text classification method and system combining SVM and semi-supervised clustering, which can improve the existing For the unbalanced text classification problem, the single classifier or algorithm classification effect is not good, and finally achieve the purpose of accurate classification of unbalanced text

Images