Document Classification Method and System Based on LDA Topic Model

Abstract

The invention discloses a document classification method based on an LDA subject model, which comprises the following steps: 1 pre-compiling an initial supervisory dictionary, wherein the words in theinitial supervisory dictionary comprise a plurality of classes and correspond to the subject of the LDA subject model one by one; 2 pre-compiling an initial supervisory dictionary. 2, obtaining all that words in the literature to be classify, calculating the probability that each word belongs to each subject, and obtaining a clustering dictionary; 3, composing a new supervision dictionary according to that cluster dictionary; 4, searching that subjects correspond to the words in the new supervised dictionary contained in each document to be classify, taking the subject with the largest numberof words as the subject of the document, and completing the classification of the documents to be classified. The invention also discloses a document classification system based on LDA subject model.The method of the invention not only maintains the accuracy of the rule classification method, but also has the associative clustering ability of LDA, the classification result is accurate, and the engineering quantity is small.

Description

technical field [0001] The invention relates to the technical field of document classification, in particular to a document classification method and system based on an LDA topic model. Background technique [0002] Existing classification methods, whether they are rule-based methods, statistical methods or deep learning methods, rely on a large amount of labeled corpus to achieve. In reality, it is very difficult to obtain all the annotated corpus, and often only part of the corpus and keywords can be determined, that is, to obtain high-precision classification results under the premise of partial prior knowledge. The technical contradiction here is reflected in the contradiction between the whole and part of the annotated corpus, between the infinite and the limited. [0003] Statistical classification with limited samples, depending on the context, will seriously damage the recall rate of classification, that is, for some obvious classification results, unexpected classi...

AI Technical Summary

Problems solved by technology

There is only one optimal cluster calculated by LDA, but there are many attributes that need to be mined in reality, so LDA cannot meet the classification requirements.

Pure LDA clustering cannot express human thoughts, so classification cannot be achieved in practice

Images