A keyword extraction method integrating language model and semantic features

Abstract

The present invention discloses a keyword extraction method integrating a language model and semantic features, comprising: extracting nouns and noun phrases in a target text to form a candidate word set, using a language model to generate a target text representation vector and a candidate word representation vector for the target text and the candidate word set respectively; performing a clustering operation on the candidate word representation vector after dimension reduction to obtain a cluster set with several clusters; determining the number of keywords to be extracted in each cluster based on the semantic similarity between the candidate word and the target text, and screening the candidate words of the corresponding number from each cluster to generate a candidate keyword set; extracting the candidate words from the candidate keyword set using a maximum margin correlation algorithm to obtain the final keyword. The present invention significantly improves the accuracy and diversity of keyword extraction.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a keyword extraction method integrating a language model with semantic features. Background Art

[0002] Keyword extraction refers to extracting a group of representative words from a document. It is a basic task in text information processing and an important topic in the field of natural language processing. Keywords extracted from documents can be regarded as a concise summary of the document. It is an important way to quickly obtain the subject content of the document. It can be used for document classification, clustering, summary generation, and recommendation, etc. It is crucial to many fields of natural language processing.

[0003] There are three main methods of keyword extraction: supervised keyword extraction, semi-supervised keyword extraction, and unsupervised keyword extraction. For supervised keyword extraction, you need to provide well-labeled training corpus, and train the keyword extraction model based on the training corpus. It is mainly divided into traditional machine learning classification extraction, sequence annotation model extraction, and statistical machine translation extraction. For semi-supervised keyword extraction, first build a keyword extraction model based on a small amount of training corpus, then use the model to extract keywords from new documents, and then manually filter and integrate the extracted keywords, add them to the training set, and retrain the model. For unsupervised keyword extraction, keywords are extracted based on text features without the need for manually labeled corpus.

[0004] Due to the high labor cost required for data annotation, the existing text keyword extraction mainly adopts unsupervised keyword extraction with strong applicability. Traditional unsupervised keyword extraction is mainly divided into three categories: keyword extraction based on statistical features, keyword extraction based on word graph model, and keyword extraction based on topic model. Keyword extraction based on statistical features mainly uses the statistical information of words in the document to extract keywords. First, the text is preprocessed to obtain a candidate word set, and then the feature value quantization method is used to screen keywords from the candidate set. Common feature values ​​include part of speech, word frequency, position information, mutual information, word span, etc., such as TF-IDF algorithm. Keyword extraction based on word graph model first constructs the language network diagram of the document, and then calculates and analyzes the words or phrases with important roles on the network diagram and uses them as keywords. For example, TextRank, PositionRank algorithm, etc. Keyword extraction based on topic model mainly uses the properties of topic distribution in the topic model to extract keywords, such as pLSA, LDA, etc. However, all kinds of traditional unsupervised keyword extraction have semantic loss problems, resulting in limited accuracy of keyword extraction in current applications. Summary of the invention

[0005] The purpose of the present invention is to provide a keyword extraction method that integrates language model and semantic features to improve the accuracy and diversity of keyword extraction.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A keyword extraction method integrating a language model and semantic features, the keyword extraction method integrating a language model and semantic features comprising:

[0008] Step 1: extract nouns and noun phrases from the target text to form a candidate word set, and use the language model to generate a target text representation vector and a candidate word representation vector for the target text and the candidate word set respectively;

[0009] Step 2: performing a clustering operation on the candidate word representation vector after dimensionality reduction to obtain a cluster set having a plurality of clusters;

[0010] Step 3: Determine the number of keywords to be extracted from each cluster based on the semantic similarity between the candidate words and the target text, and select the candidate words of the corresponding number from each cluster to generate a candidate keyword set;

[0011] Step 4: Extract candidate words from the candidate keyword set using the maximum margin correlation algorithm to obtain the final keyword;

[0012] The step of determining the number of keywords to be extracted from each cluster based on the semantic similarity between the candidate words and the target text includes:

[0013] Step 31: for each candidate word, calculate the cosine similarity between its candidate word representation vector and the target text representation vector of the target text, and use the calculated cosine similarity as the semantic weight of the candidate word;

[0014] Step 32: for each cluster in the cluster set, calculate the average value of the semantic weights of the candidate words contained in each cluster, and obtain the semantic weight of each cluster according to the average value;

[0015] Step 33: normalize the semantic weight of each cluster, use the normalized value as the semantic share of the corresponding cluster, and generate the quota of keywords to be extracted for each cluster according to the semantic share.

[0016] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0017] Preferably, the clustering operation after reducing the dimension of the candidate word representation vectors includes:

[0018] Using the UMAP dimensionality reduction algorithm, the dimension of each candidate word representation vector is reduced to obtain the candidate word representation vector after dimension reduction;

[0019] The HDBSCAN clustering algorithm is used to cluster the candidate word representation vectors after dimensionality reduction to obtain a cluster set C = {c1, c2, ..., c j , ..., c n}, where c j Represents the jth cluster in the cluster set.

[0020] Preferably, obtaining the semantic weight of each cluster according to the average value includes:

[0021] r j =t j *n j *g j

[0022] In the formula, r j is the semantic weight of the jth cluster in the cluster set, t j is the average semantic weight of the candidate words contained in the jth cluster in the cluster set, n j is the number of candidate words contained in the jth cluster in the cluster set, g i is the weight coefficient of the jth cluster in the cluster set.

[0023] Preferably, generating the quota of keywords to be extracted from each cluster according to the semantic share includes:

[0024] a j =a*f j

[0025] In the formula, a j is the number of keywords to be extracted from the jth cluster in the cluster set, a is the number of candidate words in the preset candidate keyword set to be generated, and f j is the semantic share of the jth cluster in the cluster set.

[0026] Preferably, the number a of candidate words in the preset candidate keyword set to be generated is 3 to 5 times the number of final keywords extracted.

[0027] Preferably, the method of selecting candidate words of corresponding quota from each cluster to generate a candidate keyword set includes:

[0028] According to the semantic weight of the candidate words in each cluster, the candidate words with corresponding quotas are selected from high to low, and the candidate words in each cluster are summarized to generate a candidate keyword set H = {h1, h2, ..., h q , ..., h k}, where hq Represents the qth candidate word.

[0029] The keyword extraction method integrating language model and semantic features provided by the present invention has the following beneficial effects compared with the prior art:

[0030] 1. Use pre-trained language models to extract keywords based on semantics, which improves the accuracy of new words.

[0031] 2. Through semantic clustering of candidate words, the quota of keywords to be extracted is allocated to each cluster based on the cluster semantic weight. While ensuring the comprehensiveness of the semantics of the keywords to be extracted, some non-keywords are eliminated.

[0032] 3. Candidate words are extracted through the maximum boundary correlation algorithm to ensure the accuracy and diversity of keywords. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the keyword extraction method integrating language model and semantic features of the present invention;

[0034] Figure 2 A flowchart of the target text representation and candidate word representation of the present invention;

[0035] Figure 3 Flow chart of candidate word screening of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0038] In one embodiment, in order to address the semantic loss problem existing in traditional unsupervised keyword extraction in the prior art, this embodiment, based on the existing deep learning technology, obtains text semantic vector representation based on a pre-trained language model, and studies semantic-based automatic keyword extraction algorithms through algorithms such as dimensionality reduction, clustering, and maximum marginal correlation to significantly improve the accuracy of unsupervised keyword extraction.

[0039] Specific as Figure 1As shown, the keyword extraction method integrating the pre-trained language model and the semantic features of this embodiment includes the following steps:

[0040] Step 1: Extract nouns and noun phrases from the target text to form a candidate word set, and use the language model to generate a target text representation vector and a candidate word representation vector for the target text and the candidate word set respectively. The specific steps are as follows: Figure 2 As shown:

[0041] Step 1.1, candidate word extraction: Given the target text D, perform word segmentation and part-of-speech tagging on D, extract nouns and noun phrases as candidate words, and obtain the candidate word set W = {w1, w2, ..., w i ,...,w m}, where w i Represents the i-th candidate word in the candidate word set.

[0042] Step 1.2, text vector representation: Input D into the pre-trained language model Sentence-BERT (if the length of D exceeds the longest input sequence allowed by the pre-trained language model, choose to take the specified token from the beginning to the end or the beginning and end as input according to the characteristics of the target text) to obtain the representation vector v of D D .

[0043] Step 1.3, candidate word vector representation: Input the candidate word set W into the pre-trained language model to obtain each candidate word w i The representation vector v i .

[0044] Step 2: performing a clustering operation on the candidate word representation vector after dimensionality reduction to obtain a cluster set having a plurality of clusters.

[0045] Step 2.1: Dimensionality reduction of candidate word representation vector: Use the UMAP dimensionality reduction algorithm to reduce the dimension of each candidate word w i The representation vector v i Perform dimensionality reduction to obtain the candidate word w after dimensionality reduction i The representation vector v i .

[0046] Step 2.2: Clustering of candidate word representation vectors: Using the HDBSCAN clustering algorithm, cluster the candidate words w after dimensionality reduction. i The representation vector v i Perform clustering and obtain a cluster set C = {c1, c2, ..., c j ,...,c n}, where c j Represents the jth cluster in the cluster set.

[0047] Step 3: Cluster semantic screening: Based on the semantic similarity between the candidate words and the target text, determine the number of keywords to be extracted in each cluster, and screen the candidate words of the corresponding number from each cluster to generate a set of candidate keywords.

[0048] In order to improve the accuracy of keyword extraction, this embodiment adopts a double-layer screening method to extract keywords from candidate words, wherein the candidate words are initially screened: first, the cluster semantic weight is calculated based on the semantic similarity between the candidate words and the target text in the cluster; then, based on the cluster semantic weight, the quota of keywords to be extracted in each cluster is allocated, and the candidate words are screened. The specific steps are as follows: Figure 3 As shown:

[0049] Step 31: Calculate the semantic weight of candidate words: For each candidate word w i , calculate its candidate word representation vector v i The target text representation vector v of the target text D D The cosine similarity of the word is used as the candidate word w i The semantic weight of i .

[0050] Step 32: Cluster semantic weight calculation: For each cluster c in the cluster set C j , calculate each cluster c j The average semantic weight of the candidate words contained in j , and according to the average value t j Get the semantic weight r of each cluster j , the calculation formula is as follows:

[0051]

[0052] r j =t j *n j *g j

[0053] In the formula, r j is the semantic weight of the jth cluster in the cluster set, t j is the average semantic weight of the candidate words contained in the jth cluster in the cluster set, n j is the number of candidate words contained in the jth cluster in the cluster set, g i is the weight coefficient of the jth cluster in the cluster set (used to adjust the weight distribution of cluster semantics and can be set to be equal to t j The coefficient of positive correlation, such as e tj ), s x is the candidate word w in the jth cluster x The semantic weight of x .

[0054] Step 33: normalize the semantic weight of each cluster, use the normalized value as the semantic share of the corresponding cluster, and generate the quota of keywords to be extracted for each cluster according to the semantic share.

[0055] Clustering weight normalization: In order to avoid clustering errors affecting keyword extraction, this embodiment normalizes the semantic weight r j Clusters below the specified threshold (such as 0.1) are removed. Then, the semantic weights of the remaining clusters are normalized according to the following formula to obtain the semantic share f of each cluster: j :

[0056]

[0057] In the formula, j does not take the label corresponding to the eliminated cluster.

[0058] Cluster candidate word quota: If k keywords are to be extracted from the target text D, then the number of candidate words for each cluster c is calculated. j Keyword quotaa j as follows:

[0059] a j =a*f j

[0060] In the formula, a j is the number of keywords to be extracted from the jth cluster in the cluster set. If a j If it is not an integer, it will be rounded up. a is the number of candidate words in the preset candidate keyword set to be generated. f j is the semantic share of the jth cluster in the cluster set.

[0061] Typically, the number a of candidate words in the preset candidate keyword set to be generated is 3 to 5 times the number of final keywords extracted, which is used to expand the screening range of keywords while considering the screening efficiency. This embodiment allocates quotas to each cluster according to the similarity between the candidate words and the target text, thereby achieving dynamic allocation of quotas. This ensures that the candidate words preliminarily screened are representative and that the candidate keyword set obtained by the preliminarily screened is diverse.

[0062] Cluster candidate word screening: After determining the number of clusters, each cluster c j , according to the candidate word w i The semantic weight of i , from high to low, take a j Finally, the candidate keywords under each cluster are summarized to generate a candidate keyword set H = {h1, h2, ..., h q , ..., h k}, where h qrepresents the qth candidate word, and k is the number of candidate words in the candidate keyword set.

[0063] Step 4: Use the maximum margin correlation algorithm to extract candidate words from the candidate keyword set to obtain the final keyword.

[0064] In one embodiment, the ranking score (h q ). In the MMR algorithm, the ranking score of a specific candidate word is the semantic similarity value between the candidate word and the target text minus the highest semantic similarity value between the candidate word and the selected keyword. It is used to ensure the semantic similarity between the keyword and the target text and its own diversity, which can be adjusted by parameters. Take the p candidate words with the highest ranking scores as keywords. The specific formula is as follows:

[0065]

[0066] In the formula, score(h q ) is the candidate word h in the candidate keyword set q The ranking score, sim(D, h q ) is the candidate word h q Similarity with target text D, h r is the selected keyword, R is the selected keyword set, Candidate word h q With the selected keyword r The highest value of semantic similarity, γ is the weight, which is used to adjust the relevance and diversity of extracted keywords, and can be taken as 0.5.

[0067] This embodiment performs secondary screening on the candidate keyword set obtained by the preliminary screening to ensure the accuracy of the extracted keywords, and in the secondary screening, the similarity between the current candidate word and the target text and the selected keywords is simultaneously considered to ensure that the keywords finally extracted have relevance and diversity.

[0068] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A keyword extraction method integrating language model and semantic features, characterized in that: The keyword extraction method integrating the language model and the semantic feature includes: Step 1: extract nouns and noun phrases from the target text to form a candidate word set, and use the language model to generate a target text representation vector and a candidate word representation vector for the target text and the candidate word set respectively; Step 2: performing a clustering operation on the candidate word representation vector after dimensionality reduction to obtain a cluster set having a plurality of clusters; Step 3: Determine the number of keywords to be extracted from each cluster based on the semantic similarity between the candidate words and the target text, and select the candidate words of the corresponding number from each cluster to generate a candidate keyword set; Step 4: Extract candidate words from the candidate keyword set using the maximum margin correlation algorithm to obtain the final keyword; The step of determining the number of keywords to be extracted from each cluster based on the semantic similarity between the candidate words and the target text includes: Step 31: for each candidate word, calculate the cosine similarity between its candidate word representation vector and the target text representation vector of the target text, and use the calculated cosine similarity as the semantic weight of the candidate word; Step 32: for each cluster in the cluster set, calculate the average value of the semantic weights of the candidate words contained in each cluster, and obtain the semantic weight of each cluster according to the average value; Step 33: normalize the semantic weight of each cluster, use the normalized value as the semantic share of the corresponding cluster, and generate the quota of keywords to be extracted for each cluster according to the semantic share.

2. The keyword extraction method integrating language model and semantic features as claimed in claim 1, characterized in that: The clustering operation after reducing the dimension of the candidate word representation vector includes: Using the UMAP dimensionality reduction algorithm, the dimension of each candidate word representation vector is reduced to obtain the candidate word representation vector after dimension reduction; The HDBSCAN clustering algorithm is used to cluster the candidate word representation vectors after dimensionality reduction to obtain a cluster set C = {c1, c2, ..., c j ,...,c n }, where c j Represents the jth cluster in the cluster set.

3. The keyword extraction method integrating language model and semantic features as claimed in claim 1, characterized in that: The obtaining of the semantic weight of each cluster according to the average value includes: r j =t j *n j *g j In the formula, r j is the semantic weight of the jth cluster in the cluster set, t j is the average semantic weight of the candidate words contained in the jth cluster in the cluster set, n j is the number of candidate words contained in the jth cluster in the cluster set, g i is the weight coefficient of the jth cluster in the cluster set.

4. The keyword extraction method integrating language model and semantic features as claimed in claim 1, characterized in that: The generating of the quota of keywords to be extracted from each cluster according to the semantic share includes: a j =a*f j In the formula, a j is the number of keywords to be extracted from the jth cluster in the cluster set, a is the number of candidate words in the preset candidate keyword set to be generated, and f j is the semantic share of the jth cluster in the cluster set.

5. The keyword extraction method integrating language model and semantic features as claimed in claim 4, characterized in that: The number a of candidate words in the preset candidate keyword set to be generated is 3 to 5 times the number of final keywords extracted.

6. The keyword extraction method integrating language model and semantic features as claimed in claim 1, characterized in that: The candidate words of corresponding quota are screened from each cluster to generate a candidate keyword set, including: According to the semantic weight of the candidate words in each cluster, the candidate words with corresponding quota are selected from high to low, and the candidate words in each cluster are summarized to generate the candidate keyword set H = {h1,h2,...,h q ,...,h k }, where h q Represents the qth candidate word.

Images