Method, system and storage medium for classifying text set

Abstract

The invention provides a method, system and storage medium for classifying a text set, and belongs to the technical field of the text classification algorithm. The method comprises the following steps: reading the text set needing to be classified and preprocessing the text set; determining the perplexity of the text set; determining topic number of the text set when the perplexity takes the minimum value; generating the topic vector of the text set by adopting a BTM model according to the topic number; generating a feature vector according to the text set by adopting the Doc2vec model; combining the topic vector and the feature vector to generate a feature space vector of the text set; and serving the feature space vector as the original input space vector of a SVM classifier to input into the SVM classifier, thereby performing the classification. Through the method, system and storage medium for classifying the text set disclosed by the invention, the efficiency of the text classification algorithm can be improved.

Description

technical field [0001] The present invention relates to the technical field of text classification algorithms, in particular to a method, system and storage medium for classifying text collections. Background technique [0002] Text classification algorithm is one of the commonly used algorithms in the field of computer programs. Currently, there are mainly two types of text classification algorithms. One is a classification algorithm based on machine learning methods, such as Naive Bayesian classification method based on probability statistics, decision tree method based on information entropy, k nearest neighbor algorithm, neural network classification algorithm and SVM (Support Vector Machine, Support Vector Machine) classification algorithm, etc.; the other is a classification algorithm based on deep learning, such as CNN (Convolution Neural Network, convolutional neural network) algorithm and RNN (Recurrent Neural Networks, cyclic neural network) classification algorith...

AI Technical Summary

Problems solved by technology

[0003] However, in the actual application process of the above algorithm, the problem of poor classification effect often occurs due to changes in text content and classification requirements.

For example, the Naive Bayesian classification method is limited by the assumption that the attributes that affect the classification are independent of each other, which is unrealistic; the decision tree method is not effective for the classification of data with continuous characteristics such as text; the k-nearest algorithm is widely used in text classification. The classification effect of ordinary three-layer neural network algorithm is average; SVM is a classifier with high accuracy in text classification for the same data set, but it is limited by large data sets. , the dimension of the feature space is too high when the text is represented, the computational complexity is high, and the efficiency is low

[0004] Since deep learning came into people's field of vision in 2012, many researchers have invested a lot of energy in the study of deep neural network. Deep neural network has continuously made breakthroughs in the field of image classification. The number of parameters is constantly increasing, and the computational complexity is constantly increasing. Even if someone controls the number of parameters, if a better classification effect is to be obtained, the parameters will still reach ten thousand levels.

Images