Keyword extraction system

Abstract

The invention relates to the field of natural language processing, in particular to a keyword extraction system. The system comprises a preprocessing module, a word vector conversion module, a part-of-speech tagging module, a part-of-speech screening module and a candidate word weight calculation module. According to the system, the part-of-speech of keyword extraction is limited through part-of-speech screening, then a keyword extraction direction is adjusted, and a word vector is trained by introducing a large-scale corpus library; weights of candidate words in a to-be-extracted document are calculated in a cosine distance and IF-IDF weight combination mode by depending on the word vector trained in the large-scale corpus library; and an investigation range of keyword extraction is expanded by introducing an external corpus library, so that a keyword extraction result is more reasonable and a new tool is provided for effectively extracting a keyword.

Description

technical field [0001] The invention relates to the field of natural language processing, in particular to a keyword extraction system. Background technique [0002] With the rapid development of the Internet and the advent of the era of big data, in real life, a large amount of information that humans come into contact with exists in the form of electronic documents. Faced with such a vast amount of information, people urgently need machines that can automatically identify the most Keywords that can represent the main content of the article help people understand the main content of the article faster and save people's time in reading, processing and utilizing these electronic documents. [0003] At present, this technology is called Keyword Extraction (Keyword Extraction). Keyword extraction refers to quickly obtaining multiple words or phrases that can represent the subject of the document from the document, as a refined overview of the main content of the document. Thro...

AI Technical Summary

Problems solved by technology

[0004] For the existing technology of keyword extraction, one way is to use an unsupervised method, use the statistical properties of candidate keywords, such as (TFIDF), sort them, and select the highest ones as keywords, but this way This is a pure use of statistical properties, but only uses the internal information of the document, that is, the degree of aggregation of words to discover the topic of the document

The disadvantage of this method is that the information of a document is limited, and often cannot provide enough information for discovering the subject of the document. In some documents, individual very important keywords, although the frequency of occurrence is relatively low, are important for the subject of the article. Responses play a very important role. At this time, it is impossible to extract these words only through statistical methods.

Images