Text categorization method based on probability word selection and supervision subject model

Abstract

The invention discloses a text categorization method based on probability word selection and a supervision subject model. The method includes the following steps that (1) punctuation marks in a training text are removed, word frequency information and category information are collected, and a word list and a category list are formed; (2) a subject proportion vector, a subject word matrix, a subject word distinguishing degree matrix and a regression coefficient matrix are initialized; (3) the subject proportion vector, the subject word matrix, the subject word distinguishing degree matrix and the regression coefficient matrix are updated according to the word list of the training text and category iteration of the word list; (4) for a testing text, the word frequency information is collected, and categorization is conducted by using the subject proportion vector, the subject word matrix, the subject word distinguishing degree matrix and the regression coefficient matrix. By means of the method, a complex pre-processing process during text categorization can be reduced to the maximum degree, and the testing text can be categorized more accurately. By means of the method, the distinguishing degree of words in a subject can be excavated, and the significance of the words in the text can be displayed visually.

Description

technical field [0001] The invention relates to probabilistic word selection and supervised topic models, in particular to a text classification method based on probabilistic word selection and supervised topic models. Background technique [0002] The emergence of the Internet has made it easier for people to obtain information. However, the massive data generated by the rapid development of the Internet also brings great difficulties to people's analysis and utilization of data. Therefore, it becomes more and more important to organize, manage and mine data automatically. Because of the interpretability of the underlying structure of topic models, such as PLSA (Probabilistic Latent Semantic Analysis), LDA (Latent Dirichlet Allocation), etc., they are widely used to mine low-dimensional representations of text. Topic models assume that all the words in the text are generated from a multinomial distribution called "topics", and the text is a mixture of these topics. [00...

AI Technical Summary

Problems solved by technology

Without preprocessing or improper preprocessing, the text data faced by the model will contain redundant data, which will affect the performance of the topic model

On the other hand, ignoring the different importance (or discrimination) of words in the topic relative to the identification information will also affect the performance of the topic model

Finally, supervised models based directly on word rather than topic structure cannot exploit the widespread word polysemy

Images