Mutual information based parallel feature selection method for document classification

Abstract

The present invention provides a mutual information based parallel feature selection method for document classification, which comprises: a, selecting samples and performing classification; b, solving TF-ID values of words; c, generating an initialized data set D = { x1, x2, ..., xN }; d, carrying out distributed calculating and evenly distributing all sub data sets to m calculation nodes; e, establishing sets, wherein S = phi and V = { X1, X2,..., XM }; f, calculating joint probability distribution and conditional probability distribution; g, calculating mutual information; h, selecting a feature variable; i, determining if the number is enough; and i, performing document classification. According to the parallel feature selection method for document classification, which is provided by the present invention, Rayleigh entropy based mutual information is used for measuring correlation between the feature variable and a class variable, so that the finally selected feature variable can further represent a document classification feature, a classification effect is more accurate, and a classification result is better than a result obtained by using a common feature selection method. The selection method has advantageous effects, and is suitable for promotion and application.

Description

technical field [0001] The present invention relates to a method for selecting document classification features, and more specifically, to a parallel feature selection method for document classification based on mutual information. Background technique [0002] Automatic text classification is a particularly challenging task in data analysis, both in theory and practice, and has been successfully applied in many fields, such as library literature, news paper classification, topic detection, spam filtering, author identification, web page classification Wait. With the development of information technology, in many fields, data is getting bigger and bigger, which requires more time and space. For text classification, feature selection is an important means to achieve efficient text classification without affecting the accuracy. Feature selection is a key technology to reduce dimensionality, remove irrelevant data, improve learning accuracy, and improve the intelligibility of...

AI Technical Summary

Problems solved by technology

Support vector machine is considered to be one of the most effective text classifiers, but the computing and storage resources required by support vector machine will increase rapidly with the increase of training samples, therefore, many practical problems cannot be handled by support vector machine

Images