Method and device for text classification, and method and device for characteristic processing of text classification

Abstract

The invention discloses a method and a device for text classification, and a method and a device for characteristic processing of text classification. The method for characteristic processing of text classification includes obtaining a characteristic set of learning materials used for text classification; calculating a sum of information gain values of each characteristic word in all classification types; and extracting characteristic words in a preset number in the characteristic set as learning characteristics used for text classification to enable the learning characteristics used for text classification to be part of the characteristic words in residual characteristic words except for stop words in the characteristic set, wherein the sum of the information gain values corresponding to extracted characteristic words is larger than the sum of the information gain values corresponding to non-extracted characteristic words. By applying the method and the device for text classification to characteristic extraction of text classification, noise characteristics can be effectively avoided being brought into a machine learning process, so that accuracy of text classification is improved, the scale of a characteristic library is greatly reduced, and memory usage is reduced.

Description

technical field [0001] The present application relates to the field of data processing, in particular, to a text classification method and device, and a text classification feature processing method and device. Background technique [0002] Machine learning algorithms rely on the extraction of effective feature data to obtain a good learning effect. How to extract effective features and avoid the interference of noise features is an important way to improve the effect of machine learning. [0003] At present, when obtaining the learning features of machine learning, all words are often used as features, which makes the feature library huge, thus occupying a huge amount of memory during machine learning, and it is mixed with many noise features, and the text classification effect is poor. [0004] In order to remove the noise features, the words after the stop words are deleted are used as features, but the noise features can only be eliminated to a certain extent, and the fe...

AI Technical Summary

Problems solved by technology

[0006] The main purpose of this application is to provide a text classification method and device and a text classification feature processing method and device to solve the problem that the large feature library of text classification leads to large memory usage during machine learning

Images