Text classification character screening method based on character distribution information

Abstract

The invention discloses a text classification character screening method based on character distribution information. The method is used for resolving the technical problems that an existing text classification character screening method is poor in accuracy. The technical scheme includes conducting preprocessing for each document of a document set firstly; enabling the whole document collection to be presented as a vector space modal (VSM); constructing a character dictionary; counting document frequency DF (t, Cj), comprising the character t, of each classification Ci; calculating a normalized tf*idf value of each classification Ci, and then calculating the dispersion D Intra and average inter-classification dispersion D Inter Avg of the character in each classification Ci; calculating the weight wi (t) of each character tk in each classification Ci of a text character space; and enabling all the characters to be arranged in a descending order mode according to the weight of all the characters in the whole document set, and preferentially keeping the characters having front orders during character screening. On the basis of a character distribution system, the method enables the character distribution system to be applied to the character screen process, and improves text classification efficiency and accuracy.

Description

technical field [0001] The invention relates to a text classification feature screening method, in particular to a text classification feature screening method based on feature distribution information. Background technique [0002] With the development of information and network technology, a large number of electronic documents such as news, emails, Weibo, etc. are generated every day on the Internet. As a method for efficiently classifying and managing a large number of documents, automatic text classification has been widely used in many fields. [0003] With the explosive growth of information volume, one of the main problems faced by automatic text classification is how to deal with the high-dimensional text vector feature space generated by a large amount of text data. Excessive text vector feature space will have two adverse effects on text classification methods: (1) Many mature methods cannot be optimized in high-dimensional space, and thus cannot be applied to te...

AI Technical Summary

Problems solved by technology

[0021] In order to overcome the shortcomings of the poor accuracy of existing text classification feature screening method

Images