Keyword extraction method and system based on pre-training model and automatic receptive field

Abstract

The present invention discloses a keyword extraction method and system based on a pre-training model and automatic receptive field, which belongs to the field of text keyword extraction in natural language. First, an input embedding module is constructed, the document sequence is lemmatized, and a stop word set and a mask matrix are constructed. Secondly, a receptive field and sentence vector generation network is constructed, and a receptive field matrix and a sentence vector are generated respectively. Then, the sentence vector and the receptive field matrix are used to generate a word ranking matrix, and after passing through the attention scoring layer, a candidate word score matrix is ​​obtained, and a score sequence and a length sequence are further obtained. Finally, the element pairs generated by the score sequence and the length sequence are used to complete the keyword extraction, and the network model is trained. The present invention helps the keyword extraction model to integrate the two tasks of candidate word screening and candidate word sorting, and has better robustness and adaptability.

Description

Technical Field

[0001] The present invention belongs to the technical field of text keyword extraction in natural language, and specifically relates to a keyword extraction method and system based on a pre-training model and automatic receptive field. Background Art

[0002] Large-scale pre-trained language models have significantly improved the performance of text comprehension and summarization tasks, changing the way researchers conduct research. Keyword extraction is a fundamental task in natural language processing. While this task is both versatile and popular, it also presents unique challenges.

[0003] Compared to traditional encoder-decoder models, pre-trained language models often perform better on natural language tasks. Pre-trained language models are intensively trained with massive amounts of text corpora. They incorporate sufficient broad-based text features to provide more contextual features for target sentence discrimination.

[0004] Compared to keyword generation, keyword extraction focuses more on word combinations that have appeared in the text. In keyword extraction tasks, it is common to first extract possible keyword candidates from the text based on part of speech and word segmentation. This method often has certain problems: the extraction effect depends on the accuracy of word segmentation and part of speech tagging, and is not effective for extracting words in specific fields. Moreover, the problems caused at this stage often accumulate and undermine the performance of subsequent tasks. In addition, due to the splitting of tasks, candidate word screening as an upstream task is difficult to directly integrate into the existing model. If the power of machine learning is required, it is often necessary to build another model for additional training before splicing. This method is not only cumbersome, but also cannot avoid the further accumulation of errors.

[0005] Candidate word screening based on traditional algorithms cannot meet the development requirements of modern keyword extraction technology. The pre-trained model integrates the contextual information of wide-area text and provides a flexible interface for fine-tuning. Summary of the Invention

[0006] The purpose of the present invention is to address the deficiencies in the prior art and provide a keyword extraction method and system based on a pre-training model and automatic receptive field, and to solve the problem of insufficient accuracy in candidate word screening in the keyword extraction task through the present invention. In order to solve the above problems, the present invention no longer follows the mainstream encoding-decoding model approach, but proposes a processing solution that is completely based on the pre-training model, and the keyword extraction task can be trained by fine-tuning the pre-training model. In addition, based on the flexible characteristics of the pre-training model, the present invention proposes an automatic receptive field module, and the pre-training model can give a representation of all candidate words according to the characteristics of the wide-area corpus and the target corpus itself. Through the corresponding representation, the model integrates the two steps of candidate word screening and keyword extraction into the same model. Through back-propagation training, the model can output reliable keyword extraction results.

[0007] The keyword extraction method based on the pre-trained model and automatic receptive field is implemented as follows:

[0008] Step (1) uses three preset tags [CLS], [SEP], and [PAD] to organize the original text and limit its length to obtain a document sequence, tokenize the document sequence, and then construct a stop word set and a corresponding mask matrix.

[0009] Step (2) constructs a receptive field generation network by reinterpreting the BERT (Bidirectional Encoder Representation from Transformers) pre-trained model, takes the lemmatized document sequence obtained in step (1) as input, and generates a receptive field matrix. Each element in the matrix is ​​a vector, representing an automatically generated filter, and each filter corresponds to a phrase of a specified length at the beginning of a token.

[0010] Step (3) constructs a sentence vector generation network by reinterpreting the BERT pre-training model, takes the document sequence after lemmatization in step (1) as input, and generates sentence vectors through the sentence vector generation network.

[0011] Step (4) uses the sentence vector and receptive field matrix to generate a word ranking matrix. The word ranking matrix and sentence vector are passed through a layer of attention scoring layer to obtain the candidate word score matrix.

[0012] Step (5) obtains the candidate word score matrix, compresses the candidate word score matrix, regularizes the compressed candidate word score matrix into a score sequence, and extracts the compressed position information according to the score sequence to form a length sequence.

[0013] Step (6) The length of the score sequence and the length sequence are equal, and the elements correspond one to one, generating element pairs. Obtain the preset number of keyword extractions m, and select the first m element pairs with the largest score elements in the corresponding score sequence among the element pairs; use the keyword start and length to obtain the corresponding keyword. Each element pair has its corresponding token as the beginning of the phrase. The length sequence element in the element pair represents the length of the phrase corresponding to the element pair. Use the element pair's corresponding token as the beginning of the phrase and the length sequence element in the element pair as the phrase length to obtain the phrase represented by the element pair, which is the result of keyword extraction.

[0014] Step (7) The receptive field generation network, the sentence vector generation network and the attention scoring layer constitute a keyword extraction model, and a new sequence comparison paradigm is constructed. In addition to evaluating whether the extraction results are complete, the new sequence comparison paradigm also compares the accuracy of the extraction results. The score sequence and the length sequence are accepted, and the keyword extraction model is trained through backpropagation. The reference sequence and the prediction sequence are regularized into a unified and comparable structure. The regularized sequence is used as the comparison object in the new sequence comparison paradigm. The loss function is called to calculate the difference between the two sequences, and a reliable keyword extraction model is obtained through training.

[0015] The keyword extraction system based on pre-trained models and automatic receptive fields includes: native text processing module, receptive field generation network module, sentence vector generation network module, attention scoring module, keyword extraction module and training module.

[0016] Native text preprocessing module: Processes the input native text, including using three preset markers to crop or split overly long text, organize and limit its length, obtain document sequences, tokenize them, and construct a stop word set and corresponding mask matrix.

[0017] Receptive field generation network module: takes the lemmatized document sequence as input and generates a receptive field matrix.

[0018] Sentence vector generation network module: takes the lemmatized document sequence as input and generates sentence vectors.

[0019] Attention scoring layer module: takes the receptive field matrix and sentence vector as input, and generates a candidate word score matrix by performing attention calculation on the sentence vector on the receptive field matrix.

[0020] Sequence compression module: takes the candidate word score matrix as input and generates score sequence and length sequence.

[0021] Keyword extraction module: Specify the number of keywords to be extracted. Take the score sequence and the length sequence as inputs. The score sequence and the length sequence form element pairs. Complete the keyword extraction work by sorting the scores of the element pairs. At the same time, the system supports modifying the sorting algorithm to extract keywords according to user needs.

[0022] Training module: Set whether the system enables the training mode and training parameters. In the training mode, the module accepts the score sequence and the length sequence as inputs, uses the sequence comparison paradigm, generates a new answer sequence and a new prediction sequence, and calculates the cross-entropy between the two for backpropagation.

[0023] The beneficial effects of the present invention are as follows:

[0024] The automatic receptive field pre-training model proposed by the present invention can help the keyword extraction model integrate the two tasks of candidate word screening and candidate word ranking, and use the rich semantic information in the pre-training model to extract keywords.

[0025] The results show that the present invention has better robustness and adaptability. Tested according to the standard performance index of label matching degree, the test results are all at the leading level in the field. Brief Description of the Drawings

[0026] Figure 1 is the overall implementation flowchart of the present invention;

[0027] Figure 2 is the effect example diagram of the receptive field generation network of the present invention;

[0028] Figure 3 is the overall structure schematic diagram of the present invention;

[0029] Figure 4 is the generation example diagram of the prediction sequence and the answer sequence of the present invention;

[0030] Figure 5 is the system structure schematic diagram of the present invention. Detailed Implementation Modes

[0031] First, briefly introduce the nouns involved in the embodiments of the present application:

[0032] Original text: The unprocessed text expressed in natural language form, that is, any piece of text.

[0033] Token: The smallest unit in the sequence after the text is segmented, usually represented by a number.

[0034] Stop words: Commonly used words such as "de" (of), "yi ge" (a) that are not very helpful for semantic analysis.

[0035] Attention mechanism: A commonly used algorithm in machine learning, in which the model uses automatically learned weight parameters to update values.

[0036] Encoder-Decoder: A model structure in which the encoder is responsible for converting the input into a form acceptable to the decoder, and the decoder outputs the specified target based on the conversion result.

[0037] Pre-trained model: A model that has been trained in advance and can be modified to support different tasks.

[0038] Receptive field: also refers to the convolution kernel or filter.

[0039] BERT: A pre-trained model commonly used in natural domains.

[0040] Mask: Mask operation helps data shield redundant information.

[0041] Concat: Algorithm to concatenate two tensors.

[0042] SoftMax: An algorithm that normalizes data.

[0043] Linear layer: For the input vector INPUT, perform the operation W*INPUT+bias. Assuming that the dimension of the vector INPUT is IN, the dimension of W is IN×K, and the dimension of bias is K.

[0044] The present invention will be further described below with reference to the accompanying drawings.

[0045] Reference Figure 1 This is a flow chart of the overall implementation plan of the present invention, which includes the following steps:

[0046] In step (1), the unprocessed corpus cannot be directly used by the model. For the native text to be extracted, the native text is organized using special tags and its length is limited to obtain a document sequence D. The document sequence is tokenized to obtain a tokenized document sequence D′, and then a stop word set and a corresponding mask matrix are constructed.

[0047] Step (2) takes the document sequence D′ as input, uses the BERT pre-trained model to perform feature analysis on the input, and constructs a receptive field matrix generation network. Each element in the receptive field matrix is ​​a vector, representing an automatically generated filter. Each filter corresponds to a phrase of a specified length starting with a token in the document sequence D′, and the receptive field matrix is ​​composed of filters.

[0048] Step (3) takes the document sequence D′ as input; based on another BERT pre-trained model, fine-tunes the output semantics, constructs a sentence vector generation network, and uses the document sequence D′ to output the sentence vector.

[0049] Step (4) convert the document sequence D ′ The corresponding sentence vector and receptive field matrix are taken as input; the receptive field matrix and document sequence D ′ The corresponding sentence vectors are cross-multiplied to obtain the word ranking matrix. An attention scoring layer is constructed to drive the sentence vectors to perform attention operations on the word ranking matrix to generate a candidate word score matrix.

[0050] Step (5) completes the score calculation for all word arrangements based on the candidate word score matrix, designs a compression module, takes the element with the highest probability in the candidate word score matrix according to the original sentence token, compresses it into a score sequence, and extracts the compressed position information based on the score sequence to form a length sequence.

[0051] Step (6) obtains the score sequence and length sequence, obtains the preset keyword extraction quantity m, the score sequence and length sequence contain complete keyword information, the score sequence and length sequence are equal in length, wherein the elements correspond one to one, generate element pairs, each pair of elements between the two sequences contains implicit information for generating corresponding keywords, wherein the score sequence also represents the model's evaluation of the importance of keywords. Sort the score element pairs corresponding to the score sequence of the element pairs, take the top m element pairs, each element pair has its corresponding token as the beginning of the phrase, the length sequence element in the element pair represents the length of the phrase corresponding to the element pair, use the element pair corresponding token as the beginning of the phrase, and the length sequence element in the element pair as the phrase length, all element pairs can generate the unique phrase they represent, obtain the top m element pairs of the score elements corresponding to the score sequence in the element pairs, and the phrases represented by these element pairs are the results of keyword extraction.

[0052] In step (7), the receptive field generation network, the sentence vector generation network and the attention scoring layer constitute a keyword extraction model. Since the results of keyword extraction in step (6) are easy for users to read but difficult for the keyword extraction model to use for training, a new sequence comparison paradigm needs to be constructed. In addition to evaluating whether the extraction results are complete, the new sequence comparison paradigm also compares the accuracy of the extraction results. The score sequence and the length sequence are accepted, and the keyword extraction model is trained through backpropagation. In view of the uniqueness of the keyword extraction task, the reference sequence and the prediction sequence are regularized into a unified and comparable structure. The constructed sequence is used as the comparison object in the new sequence comparison paradigm, and the loss function is called to calculate the difference between the two sequences. A reliable keyword extraction model is obtained through training.

[0053] Furthermore, the specific implementation process of step (1) is as follows:

[0054] 1-1 For the original text to be extracted, three special tags are used to organize and limit its length to obtain a document sequence. The three special tags include: [CLS] is placed before all texts; [SEP] is used to separate two sentences, and [SEP] is used to mark the end of a sentence; [PAD] is used to regularize the length of sentences for subsequent parallel calculations. Due to the limitations of the BERT pre-training model, the original sentence length needs to be limited to 512, and the document sequence D = [w1,w2,w3,…,w 512 ], where w i Represents the i-th token in the document sequence D. The token contains special marks, Chinese characters, or the smallest unit of English words after splitting. 512 ] is digitally tokenized to obtain the digitally tokenized document sequence D′ as a digital token sequence representation.

[0055] 1-2 In order to reduce the pressure on the subsequent receptive field module, the longest phrase length span is determined first. The document sequence D is used as input; a stop word set is constructed using the stop word list and part-of-speech matching rules to eliminate impossible candidate words. For conjunctions and modal particles in the document sequence D that cannot be used as the beginning of a phrase, Constructing a collection Create a matrix as shown in formula (1) Here Representing a collection The i-th element, that is, the i-th word that cannot be the beginning of a phrase, It means that in the document sequence D The corresponding element w, The dimension of the matrix is ​​512*span*512. A ternary expression is used in formula (1). Taking A=(B is a negative number)?-1:1 as an example, the ternary expression indicates that when the condition "B is a negative number" is met, A is assigned a value of -1, and when the condition "B is a negative number" is not met, A is assigned a value of 1. For punctuation marks or words such as [SEP] that cannot exist in a phrase, Constructing a collection Create a matrix as shown in formula (2) Here Representing a collection The i-th element, that is, the i-th word that cannot exist in the phrase, It means that in the document sequence D The corresponding element w, The dimension of the matrix is ​​512*span*512, and finally the two matrices and Multiply to get the complete mask matrix M T .

[0056]

[0057] Furthermore, the specific implementation process of step (2) is as follows:

[0058] 2-1 Accept document sequences D′ and M T The matrix is ​​taken as input, and the output of the hidden layer with the number of span is taken from the back to the front in the BERT output as the generated result. The output results of each hidden layer are combined and spliced, and the matrix formed by splicing the output of the hidden layer is reinterpreted as the original receptive field matrix. The original receptive field matrix is ​​passed through a thread layer to reduce the dimension, and then the result is combined with M T Matrix multiplication is performed to perform masking operations, resulting in a matrix M with a dimension of 512*span*512 filter :

[0059] M filter =LinearLayer(BERT(D′))*M T (3)

[0060] In formula (3), LinearLayer represents the linear layer, and BERT represents the BERT model.

[0061] 2-2 Constructing the mold matrix M moudule , the mold matrix is ​​essentially another Mask matrix with a dimension of 512*span*512. Its construction is shown in formula (4), where OR represents the OR operation, m ijk Represents the three-dimensional matrix M moudule The element of row j and column k in block i. Accept the matrix M filter With the mold matrix M moudule As input, the mold matrix is ​​combined with the matrix M filter Multiply and pass through the SoftMax layer as the output M′ of the receptive field generation network filter , M′ filter as the receptive field matrix.

[0062] M moudule =[m ijk =((k<i)OR(k> i+j))? 0:1] (4)

[0063] M′ filter =softmax(Mmoudule*M filter ) (5)

[0064] Figure 2This is a partial example of the receptive field matrix. The original text is a sports news item: "The two players have never played against each other before, so this match is also a battle of encounters. However, Zheng Moumou has also done some research in advance on such a strong opponent." The receptive field matrix on the right is generated. The first row represents a phrase of length one, the second row represents a phrase of length two, and so on. Each column represents the situation where a token is used as the beginning. When the length is not enough to form a phrase, that is, the element on the lower right of the matrix will be replaced by M moudule Matrix is ​​used for elimination, and the figure uses The figure uses the phrase "the first two digits" as an example to show its vector representation. Because its length is four, only the first four digits are valid and non-zero. The non-zero values ​​represent the receptive field weights generated by the model, that is, the model's weight evaluation of each character in this phrase.

[0065] Furthermore, the specific implementation process of step (3) is as follows:

[0066] 3-1 Build a new BERT model pre-training to calculate the sentence vector as the sentence vector generation network. The sentence vector generation network accepts the document sequence D′ as input, takes the output of the last four hidden layers of the BERT model, performs vector concat operation, and then passes it through the linear layer to reinterpret the concatenation result as a sentence vector. As output, where Represents the embedding vector of the i-th token, with a vector length of 768.

[0067] X=LinearLayer(concat(HiddenLayer -4: )) (6)

[0068] Furthermore, the specific implementation process of step (4) is as follows:

[0069] 4-1 The normalized receptive field matrix M′ filter Perform a cross multiplication operation with the sentence vector X to obtain the word ranking matrix M word , the matrix dimension is 512*span*768.

[0070] M word =M′ filter ×X (7)

[0071] 4-2 Construct the attention scoring layer and sort the word matrix M word And the sentence vector X passes through the linear layer to obtain two different matrices: M q ,M k . Then the two matrices M q ,M kPerform multi-head attention splitting. The dimensions of the two after multi-head attention splitting are: head*512*Dtensor, head*512*span*Dtensor. Among them, Dtensor is the dimension after 768 multi-head attention splitting. Then use the two matrices M after multi-head attention splitting q ,M k Perform attention calculation to obtain the candidate word score matrix M score , as shown in formula (8).

[0072]

[0073] Furthermore, the specific implementation process of step (5) is as follows:

[0074] 5-1 Candidate word score matrix M score It is a two-dimensional matrix with a dimension of 512*span, where each element m ij Indicates the score of the candidate word starting with the i-th token and with a length of j. score , build a screening module, select the maximum score of each column as the final score of this column, and finally compress M score The matrix is ​​a score sequence Score={s1,s2,s3,…,s 512}, where s i Indicates the maximum score obtained by the i-th token.

[0075] 5-2 Obtain score sequence Score={s1,s2,s3,…,s 512} and candidate word score matrix M score As input, take all elements in the score sequence Score, record all non-empty elements in the score sequence Score in the candidate word score matrix M score The position in the candidate word score matrix M is used to calculate the position of all elements in Score. score The column numbers of the positions in the sequence form a length sequence Length={l1,l2,l3,…,l 512}.

[0076] Furthermore, the specific implementation process of step (6) is as follows:

[0077] 6-1 Obtain the score sequence Score as input. The score sequence Score represents the final score given by the model to the i-th token, that is, the score of the keyword. By sorting all the scores, assuming the model needs to extract ten keywords, select the top ten scores from the sorted scores, record which tokens these top ten scores correspond to, and use these ten corresponding tokens to form the sequence Extract.

[0078] 6-2 takes the sequence Extract and the length sequence Length as input. Extract contains all tokens that begin with a keyword. To obtain the final keyword, for all tokens in Extract, find the lengths of these tokens in the length sequence Length. This gives all the information that constitutes the keyword. For example, for the i-th token, assuming it exists in Extract and its length in the length sequence Length is 2, the keyword corresponding to the i-th token is the phrase consisting of the i-th token and the i+1-th token. The keywords corresponding to all tokens in Extract are the result of keyword extraction.

[0079] Figure 3 It is a schematic diagram of the overall structure of the present invention; it describes the process of the model extracting keywords from native documents, wherein the Argmax process is the process of sorting the score sequence in step 6-1 and taking the highest score.

[0080] Furthermore, the specific implementation process of step (7) is as follows:

[0081] 7-1 Obtain the reference sequence provided by the training set Prefence=[a1,a2,…,a N ], where a i Represents the reference answer of the i-th keyword in the reference sequence, and uses the length of all elements in it to construct the answer sequence Answer. At the same time, for answer a i Design prediction sequence in Indicates a i The length of the phrase at the beginning of the sequence corresponds to the length of Length.

[0082] 7-2 Answer and Predict represent the length of the keywords in the reference sequence and the predicted sequence respectively. In order to more comprehensively compare whether the prediction results fully cover the reference sequence, the numbers indicating whether the keywords are predicted successfully are added to the two sequences Answer and Predict to form a complete sequence comparison paradigm. According to the score sequence Score, the sequence numbers of all elements in the Score sequence are extracted to form a set C = {c|s c >0,s c ∈Score}, where s cRepresents the elements in the score sequence with subscript c. Assuming that the total number of elements in the set C is M, all elements in C are used to index the members of the score sequence Score, and a sequence S′ with a length of M is constructed and connected to the predicted sequence Predict to obtain a new predicted sequence Predict′, as shown in formula (9), where c1, c2, …, c M Each represents an element in set C; the subscripts are used only to distinguish the elements. M represents the total number of elements in set C. Simultaneously, for all elements in S′, the function in formula (10) is called to form a sequence of the same length M. This sequence is appended to Answer to obtain a new sequence Answer′, as shown in formula (11). Predict′ and Answer′ are used as prediction and answer representations, respectively, and the loss calculation method for both constitutes a new sequence comparison paradigm.

[0083]

[0084] Figure 4 This is a specific example diagram of the generation of the Predict′ sequence and Answer′ sequence.

[0085] like Figure 5 As shown in the figure, the keyword extraction system based on the pre-trained model and automatic receptive field includes: native text processing module, receptive field generation network module, sentence vector generation network module, attention scoring module, keyword extraction module and training module.

[0086] Native text preprocessing module: This module processes native text, including using three preset markers to trim or split overly long text, organize and limit its length, obtain document sequences, and tokenize them. It also constructs a stop word set and corresponding mask matrix.

[0087] Receptive field generation network module: accepts the lemmatized document sequence as input and generates a receptive field matrix based on the mask matrix.

[0088] Sentence vector generation network module: accepts the lemmatized document sequence as input and generates sentence vectors.

[0089] Attention scoring layer module: accepts the receptive field matrix and sentence vector as input, and generates a candidate word score matrix by performing attention calculation on the sentence vector on the receptive field matrix.

[0090] Sequence compression module: accepts the candidate word score matrix as input and generates a score sequence Score and a length sequence Length.

[0091] Keyword extraction module: Specify the number of keywords to be extracted, accept the score sequence and length sequence as inputs, form element pairs from the score sequence and length sequence, and complete the keyword extraction work by sorting the scores of the element pairs. At the same time, the system supports modifying the sorting algorithm to extract keywords according to user needs.

[0092] Training module: Set whether the system enables the training mode and training parameters. In the training mode, the module accepts the score sequence and length sequence as inputs, uses the sequence comparison paradigm to generate the Predict′ and Answer′ sequences, and calculates the cross-entropy between the two for backpropagation.

[0093] Example:

[0094] Take a special sentence as the original text D: "The author of 'To Live' is Yu." and describe how the narrative model extracts keywords.

[0095] Step (1) Modify the original text using three special markers to: "[CLS] The author of 'To Live' is Yu. [SEP]", obtaining the document sequence D. Tokenize the document sequence D to obtain the tokenized document sequence D′.

[0096] Step (2) For D′, generate the receptive field matrix. Each column of the receptive field matrix represents the phrase sequence starting with the token corresponding to characters such as "[CLS], 《, 活, 着, ……, 余, 某, 。, [SEP]", etc. Each row of the receptive field matrix represents a phrase length. For example, the second row represents phrases of length two. Taking the third column of the second row as an example, it represents the phrase starting with the third token and having a length of two, which here corresponds to the phrase "活着".

[0097] Step (3) Take D ′ as the input, and use the sentence vector generation network to generate the ′ corresponding sentence vector of D.

[0098] Step (4) Take the sentence vector corresponding to D′ and the receptive field matrix as inputs; multiply the receptive field matrix and the sentence vector to obtain the word sorting matrix. Construct an attention scoring layer to drive the sentence vector to perform an attention operation on the word permutation matrix to generate the candidate word score matrix.

[0099] Step (5) According to the candidate word score matrix, compress the candidate word score matrix into a score sequence, and at the same time construct a length sequence according to the score sequence, with the sequence values as Figure 4 shown.

[0100] Step (6) Assume the number of keywords to be extracted is 3, and Figure 4Taking the score sequence Score and the length sequence length as examples, among the score sequence Score, the three highest scores are all 0.9, and the corresponding tokens are "活", "作", and "余" respectively. The values corresponding to the three tokens in the length sequence Length are all 2, indicating that the lengths of the corresponding keywords are all 2. Therefore, the keywords "活着", "作者", and "余某" are extracted.

[0101] Step (7) is as Figure 4 shown. Obtain the above-mentioned score sequence and length index sequence, and generate Predict′ using the two sequences; obtain the reference sequence {"活着", "作者", "余某"}, and generate the corresponding Answer′ sequence using the reference sequence. The number of training epochs epoch is set to 300, the learning rate is 0.00001, the training batch size batch size is 128, and the Adam optimizer is used to gradually adjust the learning rate. Calculate the cross-entropy as the loss function using Answer′ and Predict′, and perform backpropagation.

[0102] Generally, the keyword extraction task is split into two steps. First, extract keyword candidates through existing algorithms, and then achieve keyword extraction by scoring and ranking the keyword candidates. At this time, words like "活着" are usually not recognized as nouns, and non-noun phrases will lose the possibility of being keyword candidates, directly resulting in that in the subsequent process, "活着" will not participate in keyword scoring and ranking to become the extraction result. However, the present invention directly provides the implicit representation of all phrases in the original text, and uses the powerful semantic processing ability of the pre-trained model to successfully label "活着" as an important component in the sentence.

[0103] Based on the keyword extraction method of this patent invention, a comparative experiment is carried out with the existing mature keyword extraction method on the INSPEC dataset, and the INSPEC dataset is the basic dataset for keyword extraction. The evaluation metrics include: F1, P, and R. Among them, F1 represents the F1 index, P represents the accuracy rate, R represents the recall rate, and the suffix number represents the number of comparison keywords. The comparison results are shown in Table 1 below:

[0104] Table 1

[0105]

Claims

1. A keyword extraction method based on a pre-trained model and automatic receptive field, characterized in that: The specific steps are as follows: Step 1: Use preset tags to organize the raw text and limit its length to obtain a document sequence; Tokenize the document sequence and then construct a stop word set and corresponding mask matrix; Step 2: Build a receptive field generation network by reinterpreting the BERT pre-training model, take the lemma-ized document sequence as input, and generate a receptive field matrix; Step 3: Build a sentence vector generation network by reinterpreting the BERT pre-trained model, take the lemmatized document sequence as input, and generate the sentence vector through the generation network; Step 4: Use the sentence vector and receptive field matrix to generate a word ranking matrix. Pass the word ranking matrix and sentence vector through the attention scoring layer to obtain the candidate word score matrix; Step 5: Obtain the candidate word score matrix and compress the candidate word score matrix; Regularize the compressed candidate word score matrix into a score sequence, and extract the compressed position information from the score sequence to form a length sequence; Step 6: The length of the score sequence and the length sequence are equal, and the elements correspond one to one, generating element pairs; Get the preset number of keyword extractions m, and select the first m element pairs with the largest score elements in the corresponding score sequence; Use the corresponding word of the element pair as the beginning of the phrase, and the length of the sequence element in the element pair as the phrase length to obtain the phrase represented by the element pair, which is the result of keyword extraction; Step 7: The receptive field generation network, sentence vector generation network and attention scoring layer constitute a keyword extraction model, accept score sequences and length sequences, perform backpropagation training on the keyword extraction model, and obtain a reliable keyword extraction model through training.

2. The keyword extraction method based on pre-training model and automatic receptive field according to claim 1 is characterized in that: In step 1, the preset mark is as follows: [CLS]: placed before all text; [SEP]: used to separate two sentences. [SEP] is used at the end of a sentence. [PAD]: Used to regularize the length of sentences for subsequent parallel calculations.

3. The keyword extraction method based on pre-training model and automatic receptive field according to claim 1 is characterized in that: In step 2, the receptive field matrix, where each element is a vector, represents an automatically generated filter, and each filter corresponds to a phrase of a specified length at the beginning of a word.

4. The keyword extraction method based on pre-training model and automatic receptive field according to claim 3 is characterized in that: The specific process of step 2 is as follows: 2-1: The lemma document sequence D′ and the mask matrix M T As input, in the output of the BERT pre-trained model, the output results of the hidden layers with the number of span are taken from the back to the front, combined and spliced, and the spliced ​​matrix is ​​reinterpreted as the original receptive field matrix; The original receptive field matrix is ​​passed through a layer of threads to reduce the dimension, and then the result is combined with M T Multiply and perform masking operation to obtain matrix M filter : M filter =LinearLayer(BERT(D′))*M T Among them, LinearLayer represents the linear layer, and BERT represents the BERT model; 2-2: Constructing mold matrix M moudule , under the following composition: M moudule =[m ijk =((k<i)OR(k> i+j))?0:1] Among them, OR represents the OR operation, m ijk Represents the three-dimensional matrix M moudule The element in the i-th block, row j, and column k, the ternary expression means that when (k<i)OR(k> When the condition of i+j) is met, m ijk Assigned to 0, does not meet the (k<i)OR(k> i+j) condition, m ijk Assign a value of 1; Combine the mold matrix with the matrix M filter Multiply, pass through the SoftMax layer, and the receptive field generates the network output matrix M′ f ' ilter : M′ filter =softmax(M moudule *M filter ) Output matrix M′ filter is the receptive field matrix.

5. The keyword extraction method based on pre-training model and automatic receptive field according to claim 1 is characterized in that: The specific process of step 4 is as follows: 4-1: Normalized receptive field matrix M′ filter Perform a cross multiplication operation with the sentence vector X to obtain the word ranking matrix M word : M word =M′ filter ×X 4-2: Construct the attention scoring layer and sort the word matrix M word And the sentence vector X passes through the linear layer to obtain two different matrices: M q ,M k ; Then the two matrices M q ,M k Perform multi-head attention splitting and use the matrix M after two multi-head attention splits q ,M k Perform attention calculation to obtain the candidate word score matrix M score : Among them, Dtensor is the dimension after multi-head attention splitting, and concat is the vector splicing operation.

6. The keyword extraction method based on pre-training model and automatic receptive field according to claim 1 is characterized in that: In the back propagation training described in step 7, specifically, it includes: constructing a new sequence comparison paradigm, evaluating whether the extraction results are complete, and comparing the accuracy of the extraction results; regularizing the reference sequence and the predicted sequence into a unified and comparable structure, using the regularized sequence as the comparison object in the new sequence comparison paradigm, and calling the loss function to calculate the difference between the two sequences for back propagation training.

7. The keyword extraction method based on pre-training model and automatic receptive field according to claim 6 is characterized in that: The specific process of step 7 is as follows: 7-1: Get the reference sequence provided by the training set Prefence=[a1,a2,…,a N ], where a i Represents the reference answer of the i-th keyword in the reference sequence, and uses the length of all elements in it to construct the answer sequence Answer, while for a i Design prediction sequence in Indicates a i The length of the phrase at the beginning of the sequence Length; 7-2: According to the sequence numbers of all elements in the score sequence Score, a set C = {c|s c >0,s c ∈Score}, where s c Represents the elements in the score sequence with subscript c. Assuming that the total number of elements in set C is M, use all elements in C to index the members of the score sequence Score, construct a sequence S′ with a length of M and connect it to the predicted sequence Predict to obtain a new predicted sequence Predict′: Among them, c1, c2, …, c M Both represent elements in set C, and M represents the total number of elements in set C; For all elements in S′, call the function: Construct a sequence of the same length M and append it to Answer to obtain a new sequence Answer′, as shown below: Predict′ and Answer′ are used as prediction representation and answer representation respectively. The loss calculation method of the two constitutes a new sequence comparison paradigm for backpropagation training.

8. A keyword extraction system based on pre-trained models and automatic receptive fields, characterized by: include: Native text preprocessing module, receptive field generation network module, sentence vector generation network module, attention scoring module, keyword extraction module and training module; Native text preprocessing module: processes the input native text to obtain a document sequence, performs lemmatization on it, and constructs a stop word set and a corresponding mask matrix; Receptive field generation network module: takes the lemmatized document sequence as input and generates a receptive field matrix; Sentence vector generation network module: takes the lemmatized document sequence as input and generates sentence vectors; Attention scoring layer module: takes the receptive field matrix and sentence vector as input, and generates a candidate word score matrix through attention calculation; Sequence compression module: takes the candidate word score matrix as input and generates score sequence and length sequence; Keyword extraction module: specifies the number of keywords to be extracted, takes the score sequence and length sequence as input, forms element pairs from the score sequence and length sequence, and completes the keyword extraction by sorting the element pairs by scores; Training module: Set whether the system starts training mode and training parameters. In training mode, the score sequence and length sequence are used as input, and the sequence comparison paradigm is used to generate a new answer sequence and a new prediction sequence. The cross entropy of the two is calculated for back propagation.

9. The keyword extraction system based on pre-training model and automatic receptive field according to claim 8, characterized in that: The keyword extraction module also supports modifying the sorting algorithm to extract keywords in a targeted manner according to user needs.

Images