Text feature extraction method based on categorical distribution probability

Abstract

The invention discloses a text feature extraction method based on categorical distribution probability. The text feature extraction method based on the categorical distribution probability extracts text feature words by means of the manner according to which categorical distribution difference estimation is carried out on words of a text to be categorized. Mean square error values of probability distribution of each word at different categories are worked out by means of category word frequency probability of the words. A certain number of words with high mean square error values are extracted to form a final feature set. The obtained feature set is used as feature words of a text categorizing task to build a vector space model in practical application. A designated categorizer is used for training and obtaining a final category model to categorize the text to be categorized. According to the text feature extraction method based on the categorical distribution probability, category distribution of the words is accurately measured in a probability statistics manner. Category values of the words are estimated in a mean square error manner so as to accurately select features of the text. As far as the text categorizing task is concerned, a text categorizing effect of balanced linguistic data and non-balanced linguistic data is obviously improved.

Description

technical field [0001] The invention relates to computer text processing technology, in particular to a text feature extraction method based on category distribution probability. Background technique [0002] With the rapid development of the Internet, the number of electronic documents on the network has expanded rapidly. Effectively helping users find, filter and manage these massive text data has become an important content of natural language processing research. The representation of text and the selection of feature items is a basic problem in text mining and information retrieval, which quantifies the feature words extracted from the text to represent text information. Transform them from an unstructured original text into structured information that can be recognized and processed by a computer, that is, scientifically abstract the text and establish its mathematical model to describe and replace the text. It enables the computer to realize the recognition of text t...

AI Technical Summary

Problems solved by technology

[0004] At present, the use of computer technology to solve text classification problems generally adopts the vector space model, which has the problems of high word space dimension, large amount of calculation, and low classification accuracy.

Images