Keyword automatic extraction method based on distributed expression word vector calculation

Abstract

The invention relates to a keyword automatic extraction method based on distributed expression word vector calculation. The method automatically generates characteristics, and preferably solves keyword automatic extraction. The steps of the method are as followings: step 1, obtaining a training original dataset; step 2, performing preprocessing on a training set and a test text, including removing punctuation, digits, stop words, and filtering word characteristics; step 3, after the training set is obtained, through training of a linguistic model, converting the training set into a word vector table; step 4, through a distance calculation method, calculating the distance from a keyword word vector to a to-be-tested text; step 5, by different distance calculation methods, respectively obtaining arithmetic average semantic distances between distributed expression word vectors of all keywords of a field keyword set and distributed expression word vectors of all words of the test text, so as to select and sort. The method provides a new thought for extraction of keywords, semantic information of a dataset is fully used, and accuracy of automatic extraction is substantially improved.

Description

technical field [0001] The invention relates to a method for automatically extracting keywords based on distributed expression word vector calculation, belonging to the field of text mining (Text Mining). Background technique [0002] The continuous development of information technology has led to explosive growth of information in many fields, and a large amount of text information has been digitized. Electronic information resources such as digital libraries, electronic thesis databases, E-books, etc. have brought great convenience to people in collecting, storing and using information, and have become an indispensable part of modern life. With the continuous increase of electronic information, how to quickly and accurately obtain the required information from large-scale text information has become a huge challenge. Keyword extraction is an effective means to solve the above problems. It is one of the core technologies in the field of text mining and plays a very importa...

AI Technical Summary

Problems solved by technology

[0004] From the analysis of the research status at home and abroad, the current automatic keyword extraction research technology still has limitations:

[0005] (1) The existing automatic keyword extraction algorithms face many problems such as polysemous words, redundant expressions of synonyms, dynamics of thesaurus updates, and cross-domain content complexity.

[0006] (2) Most automatic keyword extraction algorithms are based on small-scale experimental samples or single documents. There is currently no good solution for keyword extraction for large-scale data sample applications, and it faces the problem of unbalanced keyword information labeling

[0007] (3) Phrases have more generalization ability than single words, and contain richer information. For practical applications, the extraction of keyword phrases is more valuable than single words, but the extraction effect of keyword phrases in current research is not good.

Images