Feature item selection and weight calculation based text classification method

Abstract

The invention discloses a feature item selection and weight calculation based text classification method, which comprises the following steps: for a corpus training set obtained through analyzing and arranging, based on the traditional TF-IDF (term frequency-inverse document frequency) feature item weight calculation, through considering the relevancy among feature items and different categories according to CHI chi-square statistics and considering the intra-category average distribution situation of the feature items according to the information entropy of the feature item, carrying out adjustment on the weights of the feature items; then, carrying out assessment and selection on the feature items according to the inter-category weight difference of the feature items; respectively expressing a text in the training set and a text to be classified into vector form by using the feature item in a feature subspace; and determining the category of the text to be classified through calculating the similarity between the text in the training set and the text to be classified. As in the processes of feature item selection and weight calculation, the inter-category and intra-category distribution situations of feature items are comprehensively considered based on TF-IDF, so that the accuracy of feature item selection is enhanced, and the dimensionality of feature items is effectively reduced, thereby improving the efficiency and accuracy of text classification.

Description

technical field [0001] The invention belongs to the technical field of text classification in information resource management, and specifically relates to a text classification method based on feature item selection and weight calculation. Background technique [0002] In the explosive growth of Internet information resources, text information is the most widely used form, because text is the carrier of information, and most other forms of information (images, sounds) can be marked with text. In order to quickly and effectively discover information and resources, text classification technology emerges as an important means to organize and manage text information effectively. [0003] Text classification is to classify texts into one or more predefined categories according to their content or attributes given a classification category. [0004] At present, the main text representation method used in the field of text classification is VSM (Vector Space Model), that is, after...

AI Technical Summary

Problems solved by technology

[0008] It can be seen that the traditional TF-IDF only considers the distribution of feature items in the text set, but ignores the distribution ratio of feature items between different text categories and the distribution of feature items within a certain category. Therefore, when using traditional TF-IDF Misjudgment may occur when classifying text based on feature items and weights

Images