Key phrase generation method and device based on pre-training model and storage medium

Abstract

The invention relates to a key phrase generation method based on a training model. The method comprises the following steps: S1, obtaining to-be-processed text data; S2, performing word segmentation and part-of-speech tagging on the acquired text data; S3, establishing a disabled lexicon, and removing words existing in the disabled lexicon; filtering out words which are not verbs and nouns; S4, performing N-gram combination to obtain a candidate word combination; S5, performing text vector conversion on the text data and the candidate word combination based on a pre-training model of Bert; S6, performing cosine similarity calculation on the vector representation of the document level and the vector representation of the candidate word, and performing semantic similarity sorting; and S7, according to a set value, selecting the words or phrases with the semantic similarity ranks in the top in the step S6 to form keywords. According to the method, the open-source pre-training model Bert is used for carrying out text vectorization expression, information of the semantic level of the text is completely obtained, keyword extraction is facilitated, phrase-level keywords are obtained according to N-gram combination, and the meaning is more complete compared with single words.

Description

technical field [0001] The invention relates to the field of text natural language processing, in particular to a key phrase generation method, device and storage medium based on a pre-trained model, which are used to identify and extract key phrase content in documents, filter redundant content, and quickly obtain valuable information. Background technique [0002] Traditional keyword extraction algorithms are divided into two categories: unsupervised methods and supervised methods. The keyword classification method based on supervised learning, such as the method, device, and Equipment and storage media, the premise of this type of method is that a large number of text predictions and corresponding keyword tags are required, and a large number of neural network model training can be used to obtain an effective model, so the acquisition cost is high. The process is cumbersome, and it is not suitable for scenarios without a large amount of labeled data and computing resource...

AI Technical Summary

Problems solved by technology

[0002] Traditional keyword extraction algorithms are divided into two categories: unsupervised methods and supervised methods. The keyword classification method based on supervised learning, such as the method, device, and Equipment and storage media, the premise of this type of method is that a large number of text predictions and corresponding keyword tags are required, and a large number of neural network model training can be used to obtain an effective model, so the acquisition cost is high. The process is cumbersome, and it is not suitable for scenarios without a large amount of labeled data and computing resources

[0003] In the unsupervised method, the keyword extraction method based on TF-IDF is commonly used. The keywords are weak, scattered or far away from the topic, so that the extracted keywords cannot express the overall semantics of the text. The existing technology such as application An invention patent with CN201710369600.9 discloses an improved TF-IDF keyword extraction algorithm and an invention patent with application number CN201410056332.1 discloses a method for automatically extracting key phrases from patent documents

Images