An Adaptive Feature Selection Method Based on Chi-Square Statistics

A feature selection method and self-adaptive technology, applied in computing, special data processing applications, natural language data processing, etc., can solve the problem of not considering the positive and negative correlation between feature items and categories, only considering the frequency of documents, zooming in on weights, etc. question

CN105512311BInactive Publication Date: 2019-02-26BEIJING UNIV OF TECH
4 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2019-02-26
Estimated Expiration
Not applicable · inactive patent

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a chi square statistic based self-adaption feature selection method and relates to the field of computer text data processing. Firstly, preprocessing of a training text set and a test text set is performed and comprises participle processing and stop word processing, then, self-adaption text feature selection based on chi square statistic is performed, a word frequency factor and interclass variance are defined and introduced into a CHI algorithm, an appropriate scaling factor is added for the CHI algorithm, finally, the scaling factor is automatically adjusted in combination of classical KNN algorithm evaluation indexes, improved CHI is adapted to different text corpora, and higher classification accuracy is guaranteed. An experimental result proves that by comparison with a conventional CHI method, the classification accuracy of a balanced corpus and a non-balanced corpus is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical field

[0001] The present invention relates to the field of computer text data processing, in particular to a method based on chi-square statistics (χ 2 , CHI) adaptive text feature selection method. Background technique

[0002] In today's era of big data, mining the potential value of data is essential. As a technology to discover the potential value of data, data mining has attracted great attention. Text data accounts for a large proportion of big data, and text classification, as a data mining method for effective organization and management of text data, has gradually become a hot spot. It is widely used in information filtering, information organization and management, information retrieval, digital libraries, and spam filtering. Text Classification (TC) refers to the process of automatically classifying unknown texts into one or more categories based on their content under a predetermined category system. Commonly used text classification methods, such as K-Ne...

Examples

Embodiment Construction

[0043] The present invention is realized by the following technical means:

[0044] An adaptive text feature selection method based on chi-square statistics. First, preprocess the training text set and test text set, including word segmentation and stop word processing. Secondly, perform adaptive text feature selection based on chi-square statistics, define word frequency factor α and inter-class variance β, and introduce them into CHI The algorithm adds an appropriate scale factor μ to the CHI algorithm. Finally, combined with the classic KNN algorithm, the scale factor μ is automatically adjusted to make the improved CHI applicable to different corpora to ensure high classification accuracy.

[0045] The above-mentioned adaptive text feature selection method based on chi-square statistics for text classification includes the following steps:

[0046] Step 1. Download the Chinese corpus released by Fudan University from the Internet-training text set and test text set;

[0047] Step...