Text classification method and system based on class-aware feature selection framework

Abstract

The invention discloses a text classification method. The method includes preprocessing a plurality of category clusters to obtain a set of feature words, respectively calculating class correlation scores and class discrimination scores between each feature word and each category cluster, and assigning each feature word to a group with Corresponding to the category cluster with the highest class correlation score, reorder the words in each category cluster, select feature subsets from each category cluster, and reorder each feature subset in the total feature set to obtain the final feature collection, and the steps of inputting the text to be classified after the vector representation into the classifier, and outputting the classification result. The data processed by the classifier in the method of the present invention also includes information such as the respective properties of different category clusters and the degree of intra-class correlation and inter-class distinction of feature words, which overcomes the one-sidedness of the prior art and can achieve better text classification. Effect. The invention is widely used in the technical field of text classification.

Description

technical field [0001] The invention relates to the technical field of text classification, in particular to a text classification method and system based on a class-aware feature selection framework. Background technique [0002] Text classification technology is widely used in practical application scenarios such as information retrieval, text mining, public opinion analysis, and spam identification. Most text classification technologies are implemented based on classifiers, and the training set used to train classifiers contains hundreds of thousands of feature words, so feature extraction is an important part of text classification technology. [0003] The purpose of feature extraction is to extract feature words that can better identify cluster categories. Existing feature extraction methods mostly extract feature words that can best identify cluster categories from a global perspective. Taking information gain as an example, its principle is to calculate the informati...

AI Technical Summary

Problems solved by technology

However, this approach does not perform well in discriminating between imbalanced datasets

This is because, when the data set has a large number of categories and is unbalanced data, the traditional feature extraction method only considers the features with the highest global class discrimination, resulting in sparse features extracted for some small sample category clusters, resulting in The classification accuracy rate for small sample clusters is reduced

At the same time, the feature extraction methods that the existing text classification methods rely on only consider the class inclination of the feature words but not the inter-class discrimination ability of the feature words. This one-sidedness limits the classification accuracy of the existing text classification methods.

Images