Text feature quantification method based on comentropy, text feature quantification device based on comentropy, text classification method and text classification device

Abstract

The invention discloses a text feature quantification method based on comentropy, a text feature quantification device based on comentropy, a text classification method and a text classification device. The text feature quantification method comprises the following steps that: the weight of each feature word in a document is calculated according to the word frequency of feature words in a text document and the comentropy distributed on different text classes; meanwhile, the inter-class distribution entropy of the feature words is calculated in different modes according to the unbalance performance of the scale of each class of a text set; in addition, the inverse document frequency is introduced as required according to the distribution features of each feature word in the text set; local word frequency factors are properly reduced, so that the weight distribution of each feature word in the document is reasonable; and the feature differences of different classes of texts are sufficiently reflected by generated document feature vectors. The text feature quantification device and the text classification device disclosed by the invention have a plurality of options or parameters; and the optimum text classification effect can be achieved through regulation. The text feature quantification method has the advantages that the text classification accuracy is improved, and the performance on different text sets is stable.

Description

technical field [0001] The invention belongs to the technical field of text mining and machine learning, and in particular relates to a text feature quantification method and device based on information entropy and a text classification method and device. Background technique [0002] The organization and mining of large-scale text data often rely on automatic text classification techniques. Automatic text classification technology generally needs to quantify the characteristics of the text before classification, so that the computer can use supervised machine learning methods for classification processing. The features of the text can be described by some words, the quantification of the text features is to select some words from the text as feature words, and assign different weights to them, so that each text document is represented by the weight value of multiple feature words. composed of vectors (called eigenvectors). It can be seen that the quantification of text fe...

AI Technical Summary

Problems solved by technology

Although some existing methods perform well on certain corpora, they perform poorly on other corpora, and their performance is unstable

Images