Chi square statistic based self-adaption feature selection method

Abstract

The invention discloses a chi square statistic based self-adaption feature selection method and relates to the field of computer text data processing. Firstly, preprocessing of a training text set and a test text set is performed and comprises participle processing and stop word processing, then, self-adaption text feature selection based on chi square statistic is performed, a word frequency factor and interclass variance are defined and introduced into a CHI algorithm, an appropriate scaling factor is added for the CHI algorithm, finally, the scaling factor is automatically adjusted in combination of classical KNN algorithm evaluation indexes, improved CHI is adapted to different text corpora, and higher classification accuracy is guaranteed. An experimental result proves that by comparison with a conventional CHI method, the classification accuracy of a balanced corpus and a non-balanced corpus is improved.

Description

technical field [0001] The invention relates to the field of computer text data processing, in particular to a method based on chi-square statistics (χ 2 ,CHI) adaptive text feature selection method. Background technique [0002] In today's era of big data, it is very important to mine the potential value of data. As a technology to discover the potential value of data, data mining has attracted great attention. Text data accounts for a considerable proportion of big data, and text classification, as a data mining method for effectively organizing and managing text data, has gradually become a hot spot. It is widely used in information filtering, information organization and management, information retrieval, digital library, and spam filtering. Text classification (Text Classification, TC) refers to the process of automatically classifying unknown texts into one or more classes according to their content under a predetermined classification system. Commonly used text cl...

AI Technical Summary

Problems solved by technology

[0004] As one of the commonly used text feature selection methods, the CHI method has the characteristics of simple implementation and low time complexity; but there are also many shortcomings, so that the classification effect is not ideal

The shortcomings of the CHI algorithm mainly include two aspects: first, CHI only considers the document frequency of feature items, and ignores the word frequency of feature items, resulting in the weight of low-frequency words bein

Images