Keyword Automatic Extraction Method Based on Distributed Expression Word Vector Calculation

Abstract

The present invention relates to a method for automatically extracting keywords based on the calculation of distributed expression word vectors. The method automatically generates features to better solve the automatic extraction of keywords. The steps are as follows: Step 1, obtaining the original training data set; Step 2 , preprocessing of the training set and test text, including: removing punctuation, numbers, stop words, and part-of-speech filtering; step 3, after obtaining the training set, convert it into a word vector table through language model training; step 4, pass the distance The calculation method is to calculate the distance from the keyword word vector to the text to be tested; Step 5, by different distance calculation methods, respectively obtain the distributed expression word vectors of all keywords in the domain keyword set to the distributed expression words of all words in the test text Arithmetic mean semantic distance of vectors for selection and sorting. This method provides a new idea for keyword extraction, can make full use of the semantic information of the dataset, and significantly improves the accuracy of automatic extraction.

Description

technical field [0001] The invention relates to a method for automatically extracting keywords based on distributed expression word vector calculation, belonging to the field of text mining (Text Mining). Background technique [0002] The continuous development of information technology has led to explosive growth of information in many fields, and a large amount of text information has been digitized. Electronic information resources such as digital libraries, electronic thesis databases, E-books, etc. have brought great convenience to people in collecting, storing and using information, and have become an indispensable part of modern life. With the continuous increase of electronic information, how to quickly and accurately obtain the required information from large-scale text information has become a huge challenge. Keyword extraction is an effective means to solve the above problems. It is one of the core technologies in the field of text mining and plays a very importa...

AI Technical Summary

Problems solved by technology

[0004] From the analysis of the research status at home and abroad, the current automatic keyword extraction research technology still has limitations:

[0005] (1) The existing automatic keyword extraction algorithms face many problems such as polysemous words, redundant expressions of synonyms, dynamics of thesaurus updates, and cross-domain content complexity.

[0006] (2) Most automatic keyword extraction algorithms are based on small-scale experimental samples or single documents. There is currently no good solution for keyword extraction for large-scale data sample applications, and it faces the problem of unbalanced keyword information labeling

[0007] (3) Phrases have more generalization ability than single words, and contain richer information. For practical applications, the extraction of keyword phrases is more valuable than single words, but the extraction effect of keyword phrases in current research is not good.

Images