Automatic Abstract Generation Method Based on Pointer Key Information

Abstract

The present invention discloses an automatic abstract generation method based on pointer key information. First, sentence vectors of multi-dimensional semantic features of an article are obtained through a pre-trained model. Secondly, pointer is used to select keyword semantic feature word vectors. Finally, an encoder is used to generate an abstract. In this method, a model framework is built using a neural network, and automated learning and training are used, eliminating excessive manual intervention. The training process adopts two methods of prediction and verification, and then the model calculates the loss and automatically improves the accuracy of the model. The method of the present invention has strong applicability and will have great applications in aspects such as news title generation and copywriting generation.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and relates to mainstream deep learning models such as the Transformer model and the Bert model. Specifically, it is a method for generating short abstract content based on pointer key information. Background Art

[0002] Abstract generation is the task of compressing a text into a short version containing the main information of the original text. Text summarization techniques can generally be divided into two extraction methods: extractive summarization, which extracts the most suitable words or sentences from the input document and summarizes them. Generative summarization, which freely generates summaries and produces new words and sentences. Extractive algorithms are easier because copying text chunks from the source document ensures grammar and accuracy. Compared with extractive methods, generative algorithms are more flexible and easier to produce smooth and coherent summaries.

[0003] In recent years, with the progress of natural language processing research, generative text summarization technology has become a hot research topic and has made remarkable progress. Rush et al., Hu et al. and Chopra et al. have proposed several Encoder-Decoder methods. Although they have achieved success, it is generally believed that the problem of insufficient representation of context semantic information in generative summaries still exists. Bahdanau et al. tried to solve this problem by using bidirectional RNN to calculate two different representations of the input sequence read from left to right and from right to left. The final vector is calculated by concatenating these two representations. The biggest problem after summary generation is out-of-vocabulary words. To address this issue, See et al. proposed the Pointer-Generator-Network (PGN), but this model has a problem of strong dependence on the vocabulary, and the generated summaries are unconstrained and uncontrollable.

[0004] The unconstrained nature of previous generative summaries leads to uncontrollable problems. First, the generated summaries contain factual errors and hallucinated content. Second, it is difficult to control the content of the summary and make the content of the summary accurately cover the key information of the article. To solve this problem, this paper proposes a model that combines keyword information with the Transformer. Using keyword information to control the generated summary to be closer to the theme, so that the generated summary can capture the key information of the article and the generated content is closer to the standard summary.

[0005] With the rapid development of deep learning technology, encouraging results have been achieved in text summarization, and the emergence of neural network models has made generative summarization technology possible. Due to the success of attention mechanisms and Transformer-based language models, pre-training-based methods have received increasing attention and achieved state-of-the-art performance in many NLP tasks. The pre-trained encoder-decoder Transformer has achieved great success in abstract generation.

[0006] In recent years, with the rapid development of deep learning, great progress has been made in natural language processing research. Especially in the field of large-scale pre-trained models, significant progress has been made by Devlin et al. and Peters et al. in pre-trained language models. In most existing generative summary models, such as BART (Bidirectional and Auto-Regressive Transformers) proposed by Lewis et al., PEGASUS (Pre-train with Extracted Gap-sentences for Abstractive Summarization) proposed by Zhang et al., and ProphetNet proposed by Qi et al., they all adopt Transformer-based architectures. Usually, they are first pre-trained in an unsupervised manner using a large corpus and then fine-tuned. These models have shown superiority in various text understanding tasks, especially in generative summarization. Although these models are effective in benchmark tests, in actual systems, problems will occur. The main reason is that they may rely heavily on spurious patterns for prediction and generate spurious content. In addition, many previous studies in language understanding have found that global semantics may be ignored by Transformer-based models. Because in these models, the self-attention mechanism is usually used to learn and predict word-level features. The sentence vectors aggregated from the word embeddings learned from existing pre-trained language models are found to be unable to effectively and sufficiently capture the semantics between sentences. This may lead to a decline in the performance of subsequent tasks. Generative summarization requires a global understanding of the source text, rather than just the meaning of individual words and sentences. Therefore, the present invention emphasizes global semantics and avoids the model from generating spurious content. In the method of the present invention, the superiority of the pre-trained model is retained, and key information is used to make up for the problem that the Transformer cannot emphasize global semantics. Summary of the Invention

[0007] In view of the deficiencies of existing abstract generation methods, the present invention proposes an automatic abstract generation method guided by keywords. In the past, keyword extraction solely used deep learning extraction algorithms or traditional keyword extraction algorithms. The method of the present invention combines the two. Since the model cannot determine which keywords have better effects, the present invention introduces pointers in PGN and uses the pointers to select the best keywords.

[0008] The automatic abstract generation method based on pointer key information of the present invention includes the following two stages:

[0009] Stage 1: Obtain an abstract generation model through training with a large amount of data;

[0010] Stage 2: Invoke the abstract generation model.

[0011] The obtaining of the abstract generation model in Stage 1 specifically includes the following steps:

[0012] 1.1) Screen and preprocess text data;

[0013] 1.2) Build a deep learning model through an encoder and a decoder;

[0014] 1.3) Train and optimize the model through a data set, including setting appropriate hyperparameters according to experience and performing multiple iterative trainings on the model;

[0015] 1.4) Obtain the abstract generation model.

[0016] The invocation of the abstract generation model in Stage 2 specifically includes the following steps:

[0017] 2.1) Package the trained model into an executable and visual program;

[0018] 2.2) Receive the input of the user;

[0019] 2.3) Preprocess the user input data, tokenize the sentences and filter characters, then classify and process and extract text features;

[0020] 2.4) Invoke the abstract generation model;

[0021] 2.5) Generate a short abstract summarizing the user input.

[0022] In the method of the present invention, the screening and preprocessing of the text data in step 1.1) includes the following operations: filtering sentences with less than 200 characters; filtering special characters in the text; splitting sentences according to punctuation marks; tokenizing according to word meanings; and converting the text into numbers with reference to a text dictionary.

[0023] In the method of the present invention, the construction of the deep learning model in step 1.2) includes the following steps: text sentence vector acquisition, TextRank key information acquisition, BiLSTM key information acquisition, pointer key information acquisition, and decoder setting.

[0024] The text sentence vector acquisition:

[0025] Encode the source file through an encoder, similar to the Transformer model. Different from only obtaining semantic information through BERT in the past, first obtain the representation information of the source text through the BERT model, and then add a self-attention block and a feed-forward block after the BERT pre-trained model to obtain the sentence vector:

[0026] x = Bert(x)

[0027] x = LN(x + SELFATTN(x))

[0028] x = LN(x + FEEDFORWARD(x))

[0029] Among them, x is the tensor representation of the input article, LN is layer normalization, SELFATTN is the self-attention network, and FEEDFORWARD is the feed-forward network.

[0030] The TextRank key information acquisition:

[0031] First, use TextRank to extract keywords from the source document, then obtain the indexes of the keywords in the original text, extract the keyword vectors from the sentence vectors according to the indexes, and then fuse the keywords extracted by BiLSTM and the keywords extracted by TextRank; TextRank is to segment the source document, then establish a graph model with each word as a node, and finally select the key information through voting; the TextRank directed graph consists of a node set V and an edge set E, and the weight between two nodes is w. For a given node v i , v i , the score is defined as follows:

[0032]

[0033] Among them, V j is the node set, v k represents the k-th node in the set V j , w ji represents the weight between the j-th node in V j and the i-th node in V i , w jk represents the set V jThe weight between the jth point and the kth point in the equation, d is the damping coefficient, and the value of d is generally 0.85. i ) is a set pointing to a given point, Out(V j ) is the set pointed to by the points, and the scores of each point are recursively calculated until convergence, and keywords are selected based on the scores.

[0034] The BiLSTM key information is obtained:

[0035] BiLSTM is used as a decoder and encoder to train the original text to obtain key information. By using parameters, the topK key information is selected (K is the number of key information to be selected);

[0036] In the BiLSTM encoding stage, the model transforms the segmented and vectorized text x into {x1,x2...,x n} is input into BiLSTM to obtain the hidden state output {h1,h2,...,h n During decoding, the unidirectional LSTM obtains the decoder’s hidden state s by reading the output of the previous timestamp. t , used to predict the current output; finally, the encoder result and the decoder hidden state are input into a linear layer and a softmax layer to obtain the attention distribution Attention distribution can be used to know which words are more important in the current prediction process. The calculation formula is as follows:

[0037] e ti =v T tanh(W h h i +W s s t )

[0038]

[0039] v T ,W h ,W s is a learnable parameter, h i is the input x i The hidden layer state, s t represents the hidden state of the decoder, e t Represents the intermediate hidden state after the linear layer. In the key information fusion stage, the pointer is used to select between the two extracted key information;

[0040] In the decoding stage, the model inputs the segmented and vectorized words into the encoder composed of LSTMs. After multiple iterations of LSTMs, multi-dimensional semantic information context of the original text is obtained. Taking context as the input, it is input into the encoder composed of LSTMs, and the top K key information with the highest scores is selected through parameter K.

[0041] Obtaining the pointer key information:

[0042] The single pointer in PGN is used to select keywords, which allows copying the keywords extracted by TextRank through the pointer and also allows copying the keywords extracted by LSTMs. Similar to the attention distribution of the pointer generation network, the attention distribution is calculated based on the keywords extracted by LSTMs as the reference keyword table:

[0043] e = vtanh(W x x + W bk b key + b attn )

[0044] a = softmax(e)

[0045] v, W x , W bk , b attn are learnable parameters, x is the word vector of the source file, and b key is the keyword extracted by BiLSTM.

[0046] The attention distribution can be regarded as the influencing factor of each keyword on the theme of the source text, which can convey information to the decoder about which keywords are more important in the subsequent process.

[0047] The weight probability is calculated from the context vector and two types of keywords:

[0048] p = σ(w x x + w bk b key + w tk t key + b p )

[0049] where w x , w bk , w tk , b p are learnable parameters, x, b key , t key are the context information, BiLSTM keywords, and TextRank keywords respectively, and σ is the sigmoid function;

[0050] The weight probability p serves as a soft switch for selecting keywords from the benchmark keyword list or from TextRank keywords. For each document, the final keyword information e is obtained by combining the two types of keywords. k :

[0051] e k = pt key + (1 - p)a.

[0052] The decoder is set as follows: The decoder must process both keywords and the source file simultaneously. The decoder consists of 6 identical layers, each layer containing four blocks, namely a multi-head attention block y1, two cross-attention blocks y2 and y3, and a feed-forward block y4, which are respectively represented as:

[0053] y1 = LN(y + MUTIHEADATTN(y))

[0054] y2 = LN(y + CROSSATTN(y, k))

[0055] y3 = LN(y + CROSSATTN(y, x))

[0056] y4 = LN(y + FEEDFORWARD(y))

[0057] Among them, y is the vector representation of the target summary. k in the first cross-attention block is the output of the keyword and the original text through the attention mechanism, and x in the second cross-attention block is the context information. Ideally, the second cross-attention block allows the model to fill in the details of the keyword information.

[0058] In stage 2 of the present invention, a visualization interface in Python is used, with an input box and an output box. After the user inputs, the model is called to generate the summary result.

[0059] The method of the present invention generates summaries in stages, with a clear and non-redundant process, avoiding scenarios that rely heavily on false patterns for prediction. Moreover, through key information, it overcomes the problem that sentence vectors learned from existing pre-trained language models cannot effectively and fully capture the semantics between sentences. In the stage of calling the model, the text content input by the user is filtered and screened in various ways to make the text more accurate and the generated summary closer to the real information. The present invention can automatically generate summary information of the text, avoiding the need for manual reading of long articles, and will have great applications in news title generation. In addition, the method of the present invention uses a simple copying mechanism to avoid the out-of-vocabulary problem and proves that keywords are very effective as guiding information for the generation of news text summaries. Brief Description of the Drawings

[0060] Figure 1It is a schematic diagram of the overall process of the method of the present invention;

[0061] Figure 2 It is a schematic diagram of the data preprocessing process in the method of the present invention;

[0062] Figure 3 It is a schematic diagram of the construction of the deep learning model in the method of the present invention;

[0063] Figure 4 It is a schematic diagram of the function of calling the abstract generation model in the method of the present invention;

[0064] Figure 5 It is a diagram showing the effect of generating an abstract in the method of the present invention. Detailed implementation manners

[0065] The following further describes the content of the present invention in conjunction with embodiments and drawings, but it is not a limitation to the present invention.

[0066] Embodiment

[0067] Refer to Figure 1 , the automatic abstract generation method based on pointer key information includes two stages:

[0068] Stage 1: Obtain an abstract generation model, including the following steps:

[0069] 1.1) Screen and preprocess text data;

[0070] 1.2) Build a deep learning model through an encoder and a decoder;

[0071] 1.3) Train the model with a data set and perform optimization;

[0072] 1.4) Obtain an abstract generation model, save the trained model in a computer, and obtain multiple types of abstract generation models by changing the model training data set or the number of iterations;

[0073] Stage 2: Call the abstract generation model, including the following steps:

[0074] 2.1) Package the trained model into an executable and visual program;

[0075] 2.2) Receive user input;

[0076] 2.3) Perform preprocessing on the user input data, tokenize the sentences and filter characters, and then classify and process to extract text features;

[0077] 2.4) Call the abstract generation model;

[0078] 2.5) Generate a short abstract that summarizes the user input.

[0079] Reference Figure 2 In step 1.1) of the present invention, text data screening and preprocessing are carried out. First, the length is preferentially filtered; then, the text that meets the standard is filtered for special characters, generally some numbers and irrelevant English characters; next, the filtered text is segmented according to punctuation marks, such as exclamation marks, full stops, question marks and other symbols indicating the end of a sentence; for each short sentence, word segmentation is carried out according to the semantic habits of Chinese, and the jieba word segmentation tool is used for word segmentation; the characters and numbers are converted, and the mapping process between characters and numbers is to read out all non-repeating characters (a single character is also called a character) in the text, and each character is marked with a digital label to form a one-to-one mapping relationship between characters and data. For example, the number "1" represents "I", and the number "2" represents "you". The present invention uses the dictionary set of the pre-trained model.

[0080] Reference Figure 3 The construction of the deep learning model includes the following steps: obtaining text sentence vectors, obtaining TextRank key information, obtaining BiLSTM key information, obtaining pointer key information, and setting the decoder.

[0081] The way to obtain text sentence vectors: use the pre-trained model and fine-tune the model parameters. The present invention selects the Bert pre-trained model, obtains the representation information of the source text through the BERT model, and then adds a self-attention block and a feed-forward block after the BERT pre-trained model to obtain the sentence vector.

[0082] x = Bert(x)

[0083] x = LN(x + SELFATTN(x))

[0084] x = LN(x + FEEDFORWARD(x)).

[0085] The TextRank model uses a graph-based method, and then obtains key information through scoring. Obtaining TextRank key information: First, use TextRank to extract keywords from the source document, then obtain the index of the keywords in the original text, extract the keyword vectors from the sentence vectors mentioned above according to the index, and then fuse the keywords extracted by BiLSTM and the keywords extracted by TextRank; TextRank is to segment the source document, then establish a graph model with each word as a node, and finally select key information through voting; the TextRank directed graph is composed of a node set V and an edge set E, and the weight between two nodes is w. For a given node v i , v i The score is defined as follows:

[0086]

[0087] Among them, V j is a set of points, and v k represents the k-th point in the set V j , w ji represents the weight between the j-th point in V j and the i-th point in V i , w jk represents the weight between the j-th point and the k-th point in the set V j . d is the damping coefficient, and the value of d is generally 0.85. In(V i ) is the set pointing to the given point, and Out(V j ) is the set pointed to by the point. Recursively calculate the scores of each point until convergence, and select keywords according to the scores.

[0088] Obtaining key information of BiLSTM: BiLSTM is used as the decoder and encoder respectively to train the original text to obtain key information. By using parameters, it is determined to select the topK key information (K determines how many key information to select);

[0089] In the BiLSTM encoding stage, the model inputs the tokenized and vectorized text x = {x1, x2..., x n} into BiLSTM and obtains the hidden state output {h1, h2,..., h n}; during decoding, the unidirectional LSTM reads the output of the previous timestamp to obtain the hidden state s t of the decoder, which is used to predict the current output; finally, the results of the encoder and the hidden state of the decoder are input into a linear layer and a softmax layer to obtain the attention distribution Through the attention distribution, it can be known which words are more important in the current prediction process. The calculation formula is as follows:

[0090] e ti = v T tanh(W h h i + W s s t )

[0091]

[0092] v T , W h , W s are learnable parameters, h i is the hidden layer state of the input x i , and s t represents the hidden state of the decoder. e tRepresents the intermediate hidden state after the linear layer. In the key information fusion stage, the pointer is used to select between the two extracted key information;

[0093] In the decoding stage, the model inputs the segmented and vectorized words into the encoder composed of LSTM. After multiple layers of LSTM iteration, the multi-dimensional semantic information context of the original text is obtained. The context is input as input to the encoder composed of LSTM, and the parameter K is used to select the key information ranked by the top K scores.

[0094] Pointer Key Information Module: Uses a single pointer in PGN to select keywords, allowing both the keywords extracted by TextRank and the keywords extracted by LSTM to be copied through the pointer. Similar to the attention distribution of the pointer generation network, the attention distribution is calculated using the keywords extracted by LSTM as the benchmark keyword table:

[0095] e=vtanh(W x x+W bk b key +b attn )

[0096] a=softmax(e)

[0097] v, W x , W bk , b attn is a learnable parameter, x is the word vector of the source file, b key The attention distribution can be viewed as the influence of each keyword on the source text topic, which can subsequently convey information to the decoder about which keywords are more important. The weighted probability is calculated using the context vector and the two keywords:

[0098] p=σ(w x x+w bk b key +w tk t key +b p )

[0099] where w x , w bk , w tk , b p are learnable parameters, x, b key , t key They are context information, BiLSTM keywords and TextRank keywords, and σ is the sigmoid function. Next, the weight probability p acts as a soft switch to select between extracting keywords from the benchmark keyword table or from the TextRank keyword. For each document, the final keyword information e is obtained by combining the two keywords.k :

[0100] e k = pt key + (1 - p)a。

[0101] Encoder settings: The decoder must process both the keywords and the source file simultaneously. The decoder consists of 6 identical layers, each layer containing four blocks, namely a multi-head attention block y1, two cross-attention blocks y2 and y3, and a feed-forward block y4, which are respectively represented as:

[0102] y1 = LN(y + MUTIHEADATTN(y))

[0103] y2 = LN(y + CROSSATTN(y, k))

[0104] y3 = LN(y + CROSSATTN(y, x))

[0105] y4 = LN(y + FEEDFORWARD(y))

[0106] Where y is the vector representation of the target summary, k in the first cross-attention block is the output of the keyword and the original text through the attention mechanism, and x in the second cross-attention block is the context information. Ideally, the second cross-attention block allows the model to fill in the details of the keyword information.

[0107] Refer to Figure 4 The process of user operation is a schematic diagram of the overall function of the text generation model called by the present invention. The text to be processed by the user is input into the input text box, click the generate summary button to obtain the summary, or click the generate key information button to obtain the keywords, and the final result is displayed in the text box. In the present invention, there corresponds to a vocabulary table, and the vocabulary table stores all the character sets that can form the training data.

[0108] To further illustrate the process of generating corresponding text according to user information, the embodiment gives specific window buttons and generated content. Refer to Figure 5 , the user inputs: a piece of news text, and then preprocesses the input text to extract the key information. Then the model is called, and the content information is input into the model, and finally a short news title is generated.

Claims

1. An automatic abstract generation method based on pointer key information, characterized in that The method comprises two stages: Phase 1: Obtaining a summary generation model, including the following steps: 1.1) Text data screening and preprocessing, including the following operations: filtering sentences with less than 200 characters; filtering special characters in the text; segmenting sentences according to punctuation; segmenting words according to their meaning; and converting text to numbers using a text dictionary; 1.2) Build a deep learning model using the encoder and decoder, including the following steps: obtaining text sentence vectors, obtaining TextRank key information, obtaining BiLSTM key information, obtaining pointer key information, and setting up the decoder; The decoder setting: The decoder must process keywords and source files simultaneously. The decoder consists of 6 identical layers, each of which contains four blocks, namely a multi-head attention block y1, two cross attention blocks y2 and y3, and a feedforward block y4, which are represented as follows: y1=LN(y+MUTIHEADATTN(y)) y2=LN(y+CROSSATTN(y,k)) y3=LN(y+CROSSATTN(y,x)) y4=LN(y+FEEDFORWARD(y)) Where y is the vector representation of the target summary, k in the first cross-attention block is the output of the attention mechanism for the keyword and the original text, and x in the second cross-attention block is the context information. The second cross-attention block allows the model to fill in the details of the keyword information. 1.3) Train the model using the dataset and perform fine-tuning; 1.4) Obtaining a summary generation model, saving the trained model in a computer, and obtaining multiple types of summary generation models by changing the model training dataset or the number of iterations; Phase 2: Calling the summary generation model, including the following steps: 2.1) Encapsulate the trained model into an executable and visual program; 2.2) Receive user input; 2.3) Preprocessing user input data: Sentence segmentation and character filtering, followed by classification and text feature extraction; 2.4) Call the summary generation model; 2.5) Generate a short summary that summarizes the user input.

2. The automatic abstract generation method based on pointer key information according to claim 1, wherein: Step 1.2) Acquire the text sentence vector: The source file is encoded through an encoder, similar to the Transformer model. Unlike the previous method of only obtaining semantic information through BERT, the BERT model is first used to obtain the representation information of the source text. Then, a self-attention block and a feedforward block are added after the BERT pre-trained model to obtain the sentence vector: x=Bert(x) x=LN(x+SELFATTN(x)) x=LN(x+FEEDFORWARD(x)) Among them, x is the tensor representation of the input article, LN is layer normalization, SELFATTN is the self-attention network, and FEEDFORWARD is the feedforward network.

3. The automatic abstract generation method based on pointer key information according to claim 2, characterized in that: Step 1.2) Acquisition of TextRank key information: First, use TextRank to extract keywords from the source document, then obtain the indexes of the keywords in the original text, extract keyword vectors from the sentence vectors according to the indexes, and then fuse the keywords extracted by BiLSTM and the keywords extracted by TextRank; TextRank is to segment the source document, then establish a graph model with each word as a node, and finally select key information through voting; The TextRank directed graph consists of a set of nodes V and a set of edges E, the weight between two nodes is w, for a given node v i , v i The score is defined as follows: Among them, V j is a set of points, and v k represents the k-th point in the set V j . w ji represents the weight between the j-th point in V j and the i-th point in V i . w jk represents the weight between the j-th point and the k-th point in the set V j . d is the damping coefficient, and the value of d is 0.

85. In(V i ) is the set of points pointing to the given point, Out(V j ) is the set of points pointed to. Recursively calculate the scores of each point until convergence, and select keywords according to the scores.

4. The automatic abstract generation method based on pointer key information according to claim 3, characterized in that: Step 1.2) Obtain BiLSTM key information: The BiLSTM is used as the decoder and encoder respectively to train the original text to obtain key information. By using parameters, it is decided to select the top-K key information, where K determines how many key information to select. In the BiLSTM encoding stage, the model inputs the tokenized and vectorized text x = {x1, x2..., x n} into the BiLSTM and obtains the hidden state outputs {h1, h2,..., h n}; during decoding, the unidirectional LSTM reads the output of the previous timestamp to obtain the hidden state s t of the decoder, which is used to predict the current output; finally, the results of the encoder and the hidden state of the decoder are input into a linear layer and a softmax layer to obtain the attention distribution The attention distribution can indicate which words are more important in the current prediction process, and the calculation formula is as follows: e ti = v T tanh(W h h i + W s s t ) v T ,W h ,W s are learnable parameters, h i is the hidden layer state of the input x i , s t represents the hidden state of the decoder, e t represents the intermediate hidden state after passing through the linear layer. In the key information fusion stage, a pointer is used to select between the two pieces of key information extracted; In the decoding stage, the model inputs the segmented and vectorized words into the encoder composed of LSTMs. Through the iteration of multiple layers of LSTMs, the multi-dimensional semantic information context of the original text is obtained. Taking context as the input, it is input into the encoder composed of LSTMs, and the top-K key information with the highest scores is selected through the parameter K.

5. The automatic abstract generation method based on pointer key information according to claim 4, wherein: The acquisition of pointer key information described in step 1.2): The single pointer in PGN is used to select keywords. It allows copying the keywords extracted by TextRank through the pointer and also allows copying the keywords extracted by LSTM. Similar to the attention distribution of the pointer generation network, the attention distribution is calculated based on the keywords extracted by LSTM as the reference keyword table: e = vtanh(W x x + W bk b key + b attn ) a = softmax(e) v, W x , W bk , b attn are learnable parameters, x is the word vector of the source file, b key is the keyword extracted by BiLSTM; The weight probability is calculated from the context vector and two types of keywords: p = σ(w x x + w bk b key + w tk t key + b p ) where w x ,w bk ,w tk ,b p are learnable parameters, x, b key ,t key are context information, BiLSTM keywords, and TextRank keywords respectively, and σ is the sigmoid function; The weight probability p serves as a soft switch for selecting keywords either from the benchmark keyword table or from TextRank keywords. For each document, the final keyword information e is obtained by combining the two types of keywords k :[[]]END]] e k = pt key + (1 - p)a。

Images