Short text-oriented optimization classification method

Abstract

The invention discloses a short text-oriented optimization classification method. The method comprises the following steps of: 1, obtaining an original data set and preprocessing the original data set; 2, selecting a feature item set from the preprocessed data set; 3, training the collected large-scale corpora by using a word vector tool to obtain a word vector model; 4, performing word vector representation on each feature item in the feature item set by using a word vector model, and performing primary clustering on the word vectors of the feature items to obtain a plurality of primary feature clusters; 5, performing two-stage loose clustering in each preliminary feature cluster to obtain a plurality of similar feature clusters; and 6, replacing the feature words obtained in the step 4 with the similar feature clusters obtained in the step 5, and then carrying out short text classification by using a classifier. Traditional short text classification mostly lacks semantic expression capability and is quite high in demnsion of the feature space; according to the invention, the semantic information of the short text can be expressed better, the dimension of the feature space is reduced, the precision and efficiency of short text classification are improved, and the short text classification method can be applied to short text classification tasks in various fields, such as spamshort message classification and microblog topic classification.

Description

technical field [0001] The invention belongs to the technical field of Chinese short text classification, and relates to an optimized classification method for short texts, in particular to a classification method for network short texts. Background technique [0002] In the information age of data explosion, the intelligence of mobile terminals and the rapid development of Internet technology have prompted people to communicate more and more frequently on the mobile Internet, resulting in a large amount of information data. Most of these data are in the form of short texts as the carrier of information transmission, such as Weibo and instant push news, etc. The content is concise, refined and rich in meaning, which has high research value. Therefore, how to automatically classify these short texts to help understand the rich meanings expressed by these short texts has become a hot and difficult research topic in the fields of natural language processing and machine learning...

AI Technical Summary

Problems solved by technology

When this method is applied to short text classification, there are the following problems: (1) When VSM calculates the semantic similarity between sentences, it does not consider the influence of synonyms in sentences on their similarity

(2) When there is a lot of text data, using VSM to represent text will cause serious dimension disaster problems

(3) Short texts are usually small in length, with many polysemous words and noisy words. The effective features of short texts extracted by traditional methods are often not enough, resulting in less semantic information representation of short texts, which is not conducive to subsequent classification

The method of using semantic expansion requires a large amount of external corpus, and also increases the computational overhead, which brings about the disaster of dimensionality, and its application scenarios are often limited

However, the method based on word vector representation alone has limited improvement in classification accuracy.

The main reason is that the word representation obtained by the traditional word embedding method or TF-IDF method only contains the semantic information or statistical information in the current text corpus, while the short text is small in length and has many polysemous words and noise words. There are fewer effective features, which makes it difficult to extract enough semantic information

Images