Incremental naive Bayes text classification method based on lifelong learning

Abstract

The invention relates to an incremental naive Bayes text classification method based on lifelong learning. The method comprises: obtaining an initial text; extracting feature words of the text, storing a feature word list of a text set and text number; calculating word frequency of each feature word in the text, dividing the text set into two parts of a training set and a validation set; through anaive Bayes classifier, training a training set vector model generated in a previous step, to obtain prior probability and feature class condition probability of the naive Bayes model and storing; ifa new text exists, training the new text in an incremental manner, and updating the prior probability and feature class condition probability of the naive Bayes model, if no new text exists, selecting a test corpus from the validation set, and according to the naive Bayes model, obtaining a prediction text category of the text corpus, and calculating accuracy rate of prediction. The method can guide learning of new tasks in an incremental manner by using knowledge learned in previous tasks, and has new feature processing ability and field adaptive ability.

Description

technical field [0001] The invention belongs to the field of data mining and machine learning, in particular to an incremental naive Bayesian text classification method based on lifelong learning. Background technique [0002] With the advent of the information age, the information we can obtain is increasing day by day, and how to process and utilize these massive data is particularly important. Although the performance of today's hardware is getting higher and higher, the amount of information is also growing explosively. Many traditional classification methods read all the data into the memory at one time when processing data, which greatly limits the generality of the algorithm. scalability and scalability. In addition, in most natural language processing tasks, the training set is incomplete. In order to improve the performance of the model, the training set needs to continuously add and update training samples. Traditional classification methods need to reintegrate a...

AI Technical Summary

Problems solved by technology

In the latter, relatively little work has been done to change the parameters of the Naive Bayesian model, but such methods are still based on the existing Naive Bayesian model, and their performance cannot be improved

Images