Short text feature extraction method

Abstract

The invention discloses a short text feature extraction method that performs feature extraction on a short text based on a knowledge base and a syntactic analysis method. The method comprises a model training process and a feature extraction process. The method comprises: performing training according to training set data; performing validation by using validation set data, and obtaining a weight set W that corresponds to a highest accuracy rate and a training model M that corresponds to the highest accuracy rate; after the feature extraction process performs processing for test set data, assigning the weight set W to each category; mapping the short text in a conceptual space by using an ESA algorithm, thereby obtaining an interpretation vector of the short text; and obtaining a topic vector through LDA, and using the vector as a final feature vector of the short text and a feature of the short text. The method provided by the invention can solve the problem that the short text is sparse in text feature and unclear in theme; and the method can reduce the difficulty in short text feature extraction processing, enhance the result of short text feature extraction, and improve accuracy of text classification.

Description

technical field [0001] The invention relates to text feature extraction and text classification methods, in particular to a short text feature extraction method. Background technique [0002] With the development of applications such as Weibo, social networking sites, and hotlines, more and more information is presented in the form of short texts, and the growth is explosive. Text mining technology can help people quickly and effectively obtain key information from massive data, and text feature extraction is a key step in text mining. [0003] Most of the existing text feature extraction methods use the method based on the Bag of Words (bag of words) model. This method usually achieves better results when used in long texts, but often does not work well when used in short texts. The main reason is that compared with long texts, short texts have the characteristics of sparse features and unclear topics. First of all, due to the limitation of the length of the short text, t...

AI Technical Summary

Problems solved by technology

[0006] A feature extraction method for short texts. This method extracts features from short texts based on the knowledge base and syntactic analysis methods. By calculating the weight of each topic, the topic vector is used as the final feature vector of the short text to solve the problem of short text feature sparsity and Issues with short texts with unclear topics; including model training process and feature extraction process

Images