Chinese text classification method based on pre-trained word vector model and random forest algorithm

Abstract

The invention provides a method for automatically classifying Chinese texts by using a pre-trained word vector model and a random forest classifier. The method comprises the following steps: traininga Word2Vec pre-training model based on an external corpus; extracting a keyword feature set of the training sample based on a TextRank algorithm, generating a word vector corresponding to each keywordin combination with a Word2Vec pre-training model, and further obtaining text features of the whole training set; performing dimension reduction on the text features of the whole training set based on a PCA principal component analysis algorithm; training a random forest classifier according to the text features and labels after dimension reduction of the whole training set; and extracting text features of the whole test set, and inputting the text features into the trained random forest classifier after dimension reduction to obtain category information of the test sample. According to the invention, the problems that an existing classification method based on machine learning has excessive redundant features, the extracted semantic information amount is limited, text semantic information is difficult to completely express, the generalization ability is insufficient, and the classification accuracy is low are solved.

Description

technical field [0001] The invention belongs to the field of natural language processing, and relates to a method for automatically classifying Chinese texts by using a pre-trained word vector model and a random forest classifier. Background technique [0002] In the era of big data, a large amount of information on the Internet is constantly pouring out. Faced with such a huge amount of data, how to allow users to quickly and effectively locate the information they need is the usual method of text classification and intelligent recommendation. Text classification technology can effectively organize and classify massive amounts of information, and use algorithmic models to classify unlabeled texts into predefined categories. Prior to this, the content of the text was generally read and then classified manually, which often required a lot of human time cost. Therefore, it is of great significance to use machine learning algorithms to automatically classify texts. Common algo...

AI Technical Summary

Problems solved by technology

[0003] However, traditional text classification methods usually take every word after text segmentation into account, which is not conducive to highlighting the importance of key words representing the text, resulting in too many redundant features

In addition, the traditional method is usually based on a certain training set for feature extraction, without the help of the huge word vector semantic information contained in the external corpus, the amount of information that can be extracted is limited, and it is not enough to fully express the semantic information of the text

Moreover, most of the traditional methods use a single machine learning algorithm without the advantages of integrated learning algorithms, so the model generalization ability is insufficient, and the accuracy of Chinese text classification is low

Images