Text keyword extraction method based on combination of Word2Vec and word co-occurrence

Abstract

The invention discloses a text keyword extraction method based on combination of Word2Vec and word co-occurrence. According to the method, an ICTCLAS word segmentation system is adopted to perform word segmentation and part-of-speech tagging on a text to obtain a vocabulary set; then, the vocabulary set is preprocessed, unreasonable vocabulary combinations are filtered out, and a preliminary candidate set is obtained; the preliminary candidate set is placed in a trained Word2Vec model to obtain a word vector table, the distance between word vectors in the word vector table is calculated, kmeans clustering is performed on the preliminary candidate set to obtain a secondary candidate set of keywords, and a word co-occurrence rate of the secondary candidate set in the preliminary candidate set is obtained according to the word vector distance; and different weight values are given to different vocabulary lengths, corresponding weights are obtained according to the word co-occurrence rateand the vocabulary lengths, ordering is performed according to the weights, and the first m keywords are final keywords. Through the method, the word vectors generated through Word2Vec are adopted toperform clustering, then the text keywords are extracted in combination with word co-occurrence and other basic characteristics, therefore, the extracted keywords are more accurate, and the method canadapt to extraction of keywords of different texts.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a text keyword extraction method based on the combination of Word2Vec and word co-occurrence. Background technique [0002] Keyword extraction is to extract some words most relevant to the meaning of this article from the text. These vocabulary can also greatly summarize the main content and central idea of ​​the article. When writing a thesis, the author generally requires to provide several keywords, which can greatly facilitate the readers to determine whether the thesis is the required one, and achieve the preview effect. [0003] Traditional keyword tagging is mainly done manually. Generally, domain experts are invited to read some specific documents, and then select some words as keywords according to the content of the text. The advantage of this is that the accuracy of keywords is relatively high, and generally fits well with the content of the arti...

AI Technical Summary

Problems solved by technology

Therefore, whether in the process of clustering or classification, the characteristics of the vocabulary cannot give sufficient information about the semantics of the vocabulary, so the accuracy of these keyword extraction algorithms is not ideal

[0006] In summary, the traditional keyword extraction method has the problems of poor keyword extraction effect and low keyword extraction efficiency.

Images