A text keyword extraction method
By combining the knowledge distillation framework of BERT, Bi-LSTM, CRF, TF-IDF, and TextRank algorithms, the keyword extraction model is optimized, and the problem of poor migration of high-quality training data sets and fields in the existing technology is solved, and the keywords are improved Accuracy of extraction and model migration.
Patent Information
- Application Number
- CN202111674485.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-31
AI Technical Summary
The existing keyword extraction method relies on high-quality training data sets and manual annotations, and the model has poor migration between different fields, and the unsupervised method has coarse statistical features, strong pattern commonality, and lacks deep semantic features, resulting in insufficient accuracy. .
Using the BERT model layer, Bi-LSTM model layer and CRF model layer combined with TF-IDF and TextRank algorithms to build a knowledge distillation framework, optimize the keyword extraction model through the fusion of Student model and Teacher model, reduce the dependence on prior knowledge and improve the The mobility and accuracy of the model.
It realizes training of larger-scale models under a small amount of data, improves the accuracy and migration of keyword extraction, and reduces the dependence of manual annotation and the cost of model training.
Smart Images

Figure CN114297388B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text processing, and in particular to a text keyword extraction method. Background Art
[0002] Text keywords represent the main concepts and key contents of the text, and are the simplest and fastest way to understand the text content. Text keyword extraction, also known as keyword extraction or keyword annotation, is an automated technology that extracts some words or phrases that are most relevant to the meaning of the text. Keyword extraction has important practical application value in the fields of library science, information science, natural language processing, etc.
[0003] Existing keyword extraction methods can be divided into two categories: supervised machine learning methods and unsupervised machine learning methods.
[0004] The general steps of the keyword extraction method based on supervised learning are: first, establish a training data set containing a large amount of text and manually annotated corresponding keywords; then, use the established training data set to train the constructed keyword extraction algorithm to obtain a model; finally, apply the trained model to extract keywords from new texts. The keyword extraction algorithm of supervised machine learning often uses machine learning models such as decision tree (DT), naive Bayes (NB), support vector machine (SVM), maximum entropy model (ME), hidden Markov model (HMM), conditional random field (CRF), etc. In the supervised keyword extraction method, the quality of the training data set will directly affect the accuracy of the model, thereby affecting the effect of keyword extraction. There are two specific reasons. On the one hand, the texts with annotated keywords are limited. It will cost a lot of manpower and material resources to annotate the training set by yourself, and the workload is very huge. In addition, the manual annotation of keywords inevitably has a certain amount of human subjective judgment, which will cause inconsistency in experimental data and affect the effect of model training. On the other hand, a keyword extraction model trained in one field is often difficult to directly apply to another field or the effect will be greatly reduced when applied to other fields. For example, a model trained on a standardized paper dataset is difficult to directly apply to Internet texts with high randomness. Therefore, how to obtain a high-quality text keyword extraction training set and make the trained keyword extraction model have good transferability between different fields is the difficulty of supervised algorithms.
[0005] Compared with supervised methods, unsupervised keyword extraction methods have been a hot topic in keyword extraction research and application in recent years. The commonly used technical means include: statistical-based methods, topic-based methods, network graph-based methods, etc.
[0006] The statistical-based methods have the following disadvantages: the feature granularity is too coarse and the pattern commonality is too strong, which makes them not universal in the task of complex representation of short texts. The statistical model lacks strong semantic features and sequence features, and simply uses cosine similarity, which lacks depth. The statistical method relies too much on prior knowledge, and the accuracy is significantly reduced when there are errors in word segmentation or part-of-speech tagging. Each word is classified independently, which does not conform to the language model because the relationship between words is not considered.
[0007] Therefore, it is necessary to use a statistical keyword extraction method that does not rely too much on prior knowledge. It can train a larger model with a smaller amount of data, make up for the various shortcomings of traditional machine learning methods, and improve the accuracy of keyword extraction. Summary of the invention
[0008] Technical issues solved
[0009] In view of the above-mentioned shortcomings of the prior art, the present invention provides a keyword extraction method, which does not rely too much on prior knowledge, can train a larger model with a smaller amount of data, and make up for the various shortcomings of traditional machine learning methods, thereby improving the accuracy of keyword extraction.
[0010] Technical Solution
[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0012] The present invention provides a text keyword extraction method, comprising the following steps:
[0013] S1. Establish a keyword extraction model, which includes a BERT model layer, a Bi-LSTM model layer and a CRF model layer. The BERT model layer is used to segment the input text, generate a number of keywords, and vectorize the keywords respectively; the Bi-LSTM model layer is used to classify the keywords and extract a category of keywords closest to the text theme; the CRF model layer is used to extract a keyword closest to the text theme from the category of keywords;
[0014] S2. Construct a training sample, wherein the training sample includes a plurality of texts, and use the training sample to train the keyword extraction model. Meanwhile, the keyword extraction model is used as a Student model, and a Teacher model is constructed respectively by using a TF-IDF algorithm and a TextRank algorithm, and a knowledge distillation framework is established, so that the Student model is close to the Teacher model, thereby optimizing the keyword extraction model.
[0015] S3. Use the trained keyword extraction model to extract text keywords.
[0016] Furthermore, the TextRank algorithm constructs the Teacher model as a graph model, and uses the graph model to obtain the score of each topic, wherein the vertices of the graph model represent topics, and the edges of the graph model represent the semantic relationship between the topics represented by the two vertices constituting the edge. The semantic relationship between the two topics determines the score weight between the two topics. The topics represented by the two vertices of the edge are denoted as ti and tj, then the score weight wi,j of the edge constituted by topics ti and tj is determined by the relative position of the candidate keywords contained in topics ti and tj, the absolute position of the candidate keywords contained in topics ti and tj, and the length of the candidate keywords contained in topics ti and tj; the topics are sorted according to the scores, and the n topics with the highest scores are selected from high to low, where n is a predefined positive integer, and the final text keywords are selected from the candidate keywords contained in the selected n topics.
[0017] Based on the same inventive concept, the present invention provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, any of the methods described above is implemented.
[0018] Based on the same inventive concept, the present invention provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.
[0019] Beneficial Effects
[0020] The present invention provides a keyword extraction method that does not rely too much on prior knowledge, can train a larger model with a smaller amount of data, and makes up for the various deficiencies of traditional machine learning methods, thereby improving the accuracy of keyword extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 A schematic diagram of the steps of a keyword extraction method provided by an embodiment of the present invention;
[0023] Figure 2 A flowchart of a keyword extraction method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] See also Figure 1-2 An embodiment of the present invention provides a text keyword extraction method, comprising the following steps:
[0026] S1. Establish a keyword extraction model, which includes a BERT model layer, a Bi-LSTM model layer and a CRF model layer. The BERT model layer is used to segment the input text, generate a number of keywords, and vectorize the keywords respectively; the Bi-LSTM model layer is used to classify the keywords and extract a category of keywords closest to the text theme; the CRF model layer is used to extract a keyword closest to the text theme from the category of keywords;
[0027] S2. Construct a training sample, wherein the training sample includes a plurality of texts, and use the training sample to train the keyword extraction model. Meanwhile, the keyword extraction model is used as a Student model, and a Teacher model is constructed respectively by using a TF-IDF algorithm and a TextRank algorithm, and a knowledge distillation framework is established, so that the Student model is close to the Teacher model, thereby optimizing the keyword extraction model.
[0028] S3. Use the trained keyword extraction model to extract text keywords.
[0029] Specifically, the statistical method uses the statistical features of the text to extract keywords, such as the frequency of keywords in the text. A relatively well-known statistical method is the TF-IDF algorithm. TF (abbreviation for "TermFrequency") represents the frequency of a candidate keyword in an article. The higher the frequency of a candidate keyword, the greater its TF value. IDF (abbreviation for "Inverse Document Frequency") represents the commonness of a candidate keyword. The more common the candidate keyword is in a document, the lower its IDF value. The main idea of the TF-IDF algorithm is that if a candidate keyword appears frequently in an article, that is, has a high TF value, and rarely appears in other articles, that is, the IDF value is also high, then this candidate keyword is very representative of the article and can be used as a keyword representing the content of the article.
[0030] Specifically, the network graph-based method is also called "graph model". It uses the network graph to rank candidate keywords, and the candidate keywords with the highest scores are selected as the keywords of the text. The TextRank algorithm is a representative algorithm among graph model algorithms. Its idea comes from the famous PageRank algorithm. It uses a voting strategy to rank candidate keywords, that is, the nodes pointed to by an important node are also more important. In the TextRank algorithm, candidate keywords within a certain window range (5 words or 10 words apart) are considered to have a semantic relationship between them and can vote for each other. After the TextRank algorithm, many models have improved it, such as the TopicRank algorithm. The TopicRank algorithm believes that the semantics of an article is represented by a topic one by one. Using a single word or phrase to represent an article is likely to cause semantic duplication of a single word or phrase. Therefore, when the TopicRank algorithm uses a graph model for sorting, the vertices of the constructed graph model are not single words or phrases, but topics clustered according to certain rules. Experiments have shown that this method can effectively improve the keyword extraction effect of the model based on the TextRank algorithm.
[0031] See also Figure 2This embodiment uses the above two methods to interfere with the keyword extraction model of the present invention to improve its accuracy. Specifically, Tf-idf and TextRank algorithms are used as Teacher Model, bert+BiLSTM+CRF is used as Student Model, and the labels and weights predicted by Teacher Model are used to train Student Model. After training for 50 Epochs, Label data is used as ground truth for another 50 Epochs of training. Then, the idea of Stacking is used to merge Teacher Model with Student Model.
[0032] In this embodiment, the TextRank algorithm constructs the Teacher model as a graph model, and uses the graph model to obtain the score of each topic, wherein the vertices of the graph model represent topics, and the edges of the graph model represent the semantic relationship between the topics represented by the two vertices constituting the edge. The semantic relationship between the two topics determines the score weight between the two topics. The topics represented by the two vertices of the edge are denoted as ti and tj, then the score weight wi,j of the edge constituted by topics ti and tj is determined by the relative position of the candidate keywords contained in topics ti and tj, the absolute position of the candidate keywords contained in topics ti and tj, and the length of the candidate keywords contained in topics ti and tj; the topics are sorted according to the scores, and the n topics with the highest scores are selected from high to low, where n is a predefined positive integer, and the final text keywords are selected from the candidate keywords contained in the selected n topics.
[0033] Based on the same inventive concept, the present invention also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the text keyword extraction method is implemented.
[0034] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor (e.g., a GPU (Graphics Processing Unit)), or other data processing chips in some embodiments. The processor is generally used to control the overall operation of the electronic device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the text keyword extraction method.
[0035] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory can also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the electronic device and its external storage device. In this embodiment, the memory is generally used to store the operating method and various application software installed in the electronic device, such as the program code of the text keyword extraction method, etc. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.
[0036] Based on the same inventive concept, the present invention also provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the text keyword extraction method is implemented.
[0037] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. A text keyword extraction method, characterized in that: The steps include: S1. Establish a keyword extraction model, which includes a BERT model layer, a Bi-LSTM model layer and a CRF model layer. The BERT model layer is used to segment the input text, generate a number of keywords, and vectorize the keywords respectively; the Bi-LSTM model layer is used to classify the keywords and extract a category of keywords closest to the text theme; the CRF model layer is used to extract a keyword closest to the text theme from the category of keywords; S2. Construct a training sample, wherein the training sample includes a plurality of texts, and use the training sample to train the keyword extraction model. Meanwhile, the keyword extraction model is used as a Student model, and a Teacher model is constructed respectively by using a TF-IDF algorithm and a TextRank algorithm, and a knowledge distillation framework is established, so that the Student model is close to the Teacher model, thereby optimizing the keyword extraction model. S3, extracting text keywords using the trained keyword extraction model; Use Tf-idf and TextRank algorithms as Teacher Model, use bert+BiLSTM+CRF as Student Model, use the labels and weights predicted by Teacher Model to train Student Model, and after training for 50 epochs, use Label data as ground truth for another 50 epochs. Then use the idea of Stacking to merge Teacher Model with Student Model.
2. The text keyword extraction method according to claim 1, characterized in that: The TextRank algorithm constructs the Teacher model as a graph model, and uses the graph model to obtain the score of each topic, wherein the vertices of the graph model represent topics, and the edges of the graph model represent the semantic relationship between the topics represented by the two vertices constituting the edge. The semantic relationship between the two topics determines the score weight between the two topics. The topics represented by the two vertices of the edge are denoted as ti and tj, then the score weight wi,j of the edge constituted by topics ti and tj is determined by the relative position of the candidate keywords contained in topics ti and tj, the absolute position of the candidate keywords contained in topics ti and tj, and the length of the candidate keywords contained in topics ti and tj; the topics are sorted according to the scores, and the n topics with the highest scores are selected from high to low, where n is a predefined positive integer, and the final text keywords are selected from the candidate keywords contained in the selected n topics.
3. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to claim 1 or 2 is implemented.
4. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to claim 1 or 2 is implemented.
