Text classification method based on chi square statistics and SMO algorithm

Abstract

The invention discloses a text classification method based on chi square statistics and an SMO algorithm. The method comprises the steps that first, training texts are subjected to word segmentation, stop word removing and preprocessing, and then a chi square statistics quantity is used as a standard for selecting a set number of words to be used as feature words; then, the feature weight values of the training texts and testing texts are computed respectively; feature vectors of each training text and each testing text are converted into training document vector models and testing document vector models; and a trained classifier carries out classification on the feature vectors of the testing texts, and the classifying result of each testing text is obtained. According the method, the shortcomings that a lot of text classification features and a lot of noise exist due to the fact that all words are used as features can be overcome, and text classification accuracy and efficiency can be improved.

Description

technical field [0001] The invention relates to the technical field of natural language computer automatic processing, in particular to a text classification method based on chi-square statistics and SMO algorithm. Background technique [0002] In recent years, with the rapid development and popularization of Internet technology, the electronic resource information on the network has increased dramatically. Facing such a large amount of data information, how to effectively organize and manage these massive information, and obtain the information you need quickly and accurately? Information of real interest has become a major problem at present. Most of the network information is stored in the form of text, so the mining of text data has high potential value. As a typical text mining technology, text classification technology can organize and process a large amount of text information, facilitate information retrieval and analysis, and facilitate users to quickly and accurat...

AI Technical Summary

Problems solved by technology

[0007] The main purpose of the present invention is to provide a text classification method based on chi-square statistics and SMO algorithm for the deficiencies in the prior art, which can overcome the defects that the text classification has many features and many noises caused by using all words as features. And can improve the accuracy and efficiency of text classification

Images