LDA-based text classification method

Abstract

The invention provides an LDA-based text classification method. The method comprises the following steps of: extracting and inputting a feature word set into a text classification model so as to calculate the probability of each type in A predetermined types to which a text belongs, and taking the type with the maximum probability value as the type to which the text belongs; previously training an LDA topic model by using a training corpus according to a set topic number K, so as to K topic associated word sets; previously verifying the text classification model by using a type-specific verification corpus, so as to obtain a classification correctness of each type in the A types; when classification is carried out by using the text classification model, directly outputting a result if the classification correctness, obtained by the text classification model, of the type achieves a set threshold value; and otherwise, calculating the weighted values of K topics corresponding to the text by using the LDA topic model, selecting the topic with the maximum weighted value, forming an expanded feature word set by the first Y words in the associated words of the topic, and carrying out classification again by using the text classification model. The method provided by the invention is strong in scene adaptability and high in result usability.

Description

technical field [0001] The invention relates to a text classification method based on LDA. Background technique [0002] Text classification technology is the core technology in the field of information retrieval and data mining. The main algorithms include Bayesian, K nearest neighbor, neural network and SVM. Among them, the Bayesian algorithm assumes that the features are independent of each other when performing text classification, which greatly simplifies the training and classification process, so it has the characteristics of fast operation and easy implementation, and has become widely used in text classification. A method that has attracted the attention of many scholars. Someone proposed a naive Bayesian text classification algorithm based on expectation maximization (EM), which improves the utilization rate of unlabeled corpus. Some people combine the naive Bayesian text classification algorithm with the SVM algorithm to improve the classification accuracy. How...

AI Technical Summary

Problems solved by technology

However, with the discovery of the Internet, the Internet is filled with a large number of information texts in various forms such as news, blogs, and meeting minutes. Such information texts more or less include academic-related information, and often include the latest Academic research information is concerned by people from various disciplines, and such information texts are disorderly, often overlapping disciplines, and generally do not have classification information themselves; it is often difficult to make correct automatic classification of such information texts in existing technologies

Images