Document semantic representation method based on thematic word class similarities and text classification method and device

Abstract

The invention relates to a document semantic representation method based on thematic word class similarities and a text classification method and device. The document semantic representation method based on thematic word class similarities comprises the steps that (1) a word vector model is used to train a corpus, and word vectors are obtained; (2) clustering is performed on the word vectors in asemantic space; and (3) a WMD algorithm is used to calculate the distance between a to-be-represented document and each category obtained through clustering, and the obtained distance is used as semantic representation of the to-be-represented document. Then document classification is realized by calculating the similarities among semantic representation vectors of the document. Through the document semantic representation method, according to semantic information, word frequency and other information of texts, transfer cost between text words and clustering sets is calculated through the WMDmodel, each text is represented with one low-dimensional dense vector containing the semantic information, text information can be better represented, the classification task is high in accuracy, andthe method can be applied to natural language processing tasks such as information retrieval and text classification.

Description

technical field [0001] The invention belongs to the field of information technology, and in particular relates to a document semantic representation method, a text classification method and a corresponding device based on the similarity of subject parts of speech. Background technique [0002] Text vector representation is one of the key technologies in the fields of text mining and natural language processing. A good document semantic representation method can improve the performance of tasks such as information retrieval and text classification. [0003] The present invention is a document semantic representation method based on the similarity of subject parts of speech, and is an improvement proposed for the high-dimensional sparseness and no semantics of the bag-of-words model. At present, the document representation methods based on the bag-of-words model are as follows: [0004] 1) The traditional bag of words (Bag of words, BOW) model represents the frequency of words...

AI Technical Summary

Problems solved by technology

[0007] There are some deficiencies in the existing bag-of-words model document representation methods, in which the BOW model only considers the frequency of words, not the semantic information of words; the TF-IDF model represents the text as a vector through the combination of word frequency and inverse document frequency, and does not Considering the semantic information of the text, it is easy to suffer from the problem of data sparsity; while the FBOW model uses the positional relationship of words in the semantic space to represent the correlation between words, and does not represent the semantic information represented by the document as a whole

Therefore, the document semantic vector representation method still has a lot of room for improvement.

Images