A text feature extraction method based on inter-class discrimination and intra-class high representation

Abstract

The invention discloses a text feature extracting method based on inter-class distinctness and intra-class high representation degree. The method comprises the following steps: preprocessing a training set text; calculating the class distinctness of each feature word through an improved feature selecting method so as to select feature words with more class representation, wherein the selected feature words are of high distinctness among different classes; and further screening the selected feature words which are of high class distinctness based on the intra-class distribution rate and information gain (IG) of the feature words. With the adoption of the method, the feature selection is carried out twice to select the feature words which are of high intra-class information entropy and high intra-class distribution rate, and thus the classifying efficiency and accuracy can be improved; in addition, the calculation is simple, so that the text classifying speed and accuracy can be improved.

Description

technical field [0001] The invention belongs to the technical field of text mining, and in particular relates to a text feature extraction method based on inter-category discrimination and intra-class high representation. Background technique [0002] In today's era of rapid growth of Internet information resources, in order to find the required information and resources more quickly and effectively, text classification technology has emerged as an important means of effectively organizing and managing text information. Text classification technology refers to the technology of classifying texts to be processed into one or more predefined categories according to their content or attributes. In the field of text classification, it is currently popular to use VSM vector space to represent text. In order to avoid the "high-dimensional disaster" of feature items generated when VSM space is established, the feature selection algorithm becomes particularly important. [0003] The...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the problem that the existing text classification feature selection method cannot select the feature words with high category representativeness and the large amount of calculation, the present invention provides a method based on inter-category discrimination and intra-category high representation, and the calculation amount is small The text feature extraction method of

Images