Feature selection method based on entry distribution

Abstract

The invention discloses a feature selection method based on entry distribution. The method includes the following steps that a plurality of texts different in class are collected to form a data set, all the texts in the data set are preprocessed, and a plurality of entries are obtained; the class average word frequency of each entry in the corresponding class is calculated; the overall average word frequency of each entry in the whole data set is calculated; the weight value of each entry is calculated according to the class average word frequency and the overall average word frequency of the corresponding entry; all the entries are arranged in a descending mode according to the weight values; the front 5-30 percent of the entries in the data set are selected to serve as feature words used for classifying the texts. According to the feature selection method based on entry distribution, the word frequency serves as the basis, and the classifying precision can be improved.

Description

technical field [0001] The invention belongs to the technical field of data mining and relates to a feature selection method based on entry distribution. Background technique [0002] With the rapid growth of network information, the number of electronic texts has increased dramatically, how to effectively organize these resources has attracted more and more researchers' attention. Text classification is the key technology to solve this problem. Text classification is to use some marked texts to construct classifiers, and then automatically classify unmarked texts into predefined categories according to the constructed classifiers. This technology is widely used in Web text classification, information retrieval, mail filtering and spam SMS filtering and other fields. [0003] At present, there are a large number of classification algorithms, such as decision tree, k-Nearest Neighbors (kNN, k-Nearest Neighbors), Support Vector Machine (SVM, Support Vector Machine) and so on...

AI Technical Summary

Problems solved by technology

However, word frequency also has a strong influence on feature selection during classification

Therefore, someone proposed the t-test method. Although this method is also based on word frequency, the classification accuracy is not high because the interaction between categories in the data set is not well considered.

Images