An Optimal Classification Method for Short Texts

Abstract

The invention discloses an optimized classification method for short texts. Step 1: Obtain an original data set and preprocess it; Step 2: Select a feature item set from the preprocessed data set; The vector tool trains the collected large-scale corpus to obtain the word vector model; Step 4: Use the word vector model to represent each feature item in the feature item set as a word vector, and perform a preliminary clustering of the word vectors of the feature items to obtain Several preliminary feature clusters; Step 5: Perform two-stage loose clustering within each preliminary feature cluster to obtain several similar feature clusters; Step 6: Replace the feature words obtained in Step 4 with the similar feature clusters obtained in Step 5 , and then use the classifier for short text classification. Most of the traditional short text classification lacks semantic expression ability, and the dimension of the feature space is relatively high. The present invention can better express the semantic information of the short text while reducing the dimension of the feature space, thereby improving the accuracy and efficiency of short text classification. It is used in short text classification tasks in various fields, such as spam SMS classification, Weibo topic classification, etc.

Description

technical field [0001] The invention belongs to the technical field of Chinese short text classification, and relates to an optimized classification method for short texts, in particular to a classification method for network short texts. Background technique [0002] In the information age of data explosion, the intelligence of mobile terminals and the rapid development of Internet technology have prompted people to communicate more and more frequently on the mobile Internet, resulting in a large amount of information data. Most of these data are in the form of short texts as the carrier of information transmission, such as Weibo and instant push news, etc. The content is concise, refined and rich in meaning, which has high research value. Therefore, how to automatically classify these short texts to help understand the rich meanings expressed by these short texts has become a hot and difficult research topic in the fields of natural language processing and machine learning...

AI Technical Summary

Problems solved by technology

When this method is applied to short text classification, there are the following problems: (1) When VSM calculates the semantic similarity between sentences, it does not consider the influence of synonyms in sentences on their similarity

(2) When there is a lot of text data, using VSM to represent text will cause serious dimension disaster problems

(3) Short texts are usually small in length, with many polysemous words and noisy words. The effective features of short texts extracted by traditional methods are often not enough, resulting in less semantic information representation of short texts, which is not conducive to subsequent classification

The method of using semantic expansion requires a large amount of external corpus, and also increases the computational overhead, which brings about the disaster of dimensionality, and its application scenarios are often limited

However, the method based on word vector representation alone has limited improvement in classification accuracy.

The main reason is that the word representation obtained by the traditional word embedding method or TF-IDF method only contains the semantic information or statistical information in the current text corpus, while the short text is small in length and has many polysemous words and noise words. There are fewer effective features, which makes it difficult to extract enough semantic information

Images