Text summarization generation method based on keyword information and multi-head attention mechanism

Abstract

The application relates to the technical field of natural language processing, and discloses a text abstract generation method based on keyword information and a multi-head attention mechanism, which takes a classical seq2seq network model with an attention mechanism as a main body, and adds a keyword acquisition branch, the keyword acquisition branch is used for keyword extraction and filtering on input text to acquire refined keywords, and the refined keywords are input into an attention mechanism module of the classical seq2seq network model for subsequent abstract generation; Chinese texts are collected, a data set is constructed, an abstract generation model is trained, and then the trained abstract generation model is used for automatic abstract generation on to-be-inspected texts. The method has a remarkable effect on the improvement of the text abstract effect, and the abstract effect example shows that the model has good readability and pays attention to keyword information of the original text.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to a text abstract generation method based on keyword information and a multi-head attention mechanism. BACKGROUND

[0002] With the development of deep learning, a large number of automatic text abstract models are constantly emerging. At present, common text abstract models are mainly based on sequence-to-sequence type models. However, the existing models often have problems such as inaccurate semantic information and out-of-vocabulary generation words.

[0003] According to the automatic abstract generation mode, automatic text abstract can be divided into extractive text abstract and generative text abstract. The extractive abstract generates the final abstract by extracting sentences from the original text content and combining them. The extractive abstract has the characteristics of ensuring the continuity of the sentence, low grammatical and syntactic error rate, and stable abstract accuracy. However, the extractive abstract has the defects of sentence repetition and poor connection between sentences, resulting in poor readability. The generative abstract solves this problem. Compared with the extractive abstract, the generative abstract is more flexible. It mainly extracts words from the constructed word table to combine the final generated abstract, and allows different words or phrases in the text from the original text, so as to generate a more personalized abstract result according to the text content. However, the performance of the generative abstract in long text is not good. The recurrent neural network (RNN) and long short-term memory network (LSTM) commonly used in the encoding layer and decoding layer have long-distance dependence problem, which limits the performance of the generative abstract. In the transmission process, part of the information is easily lost, and the key information is often lost, the sentence continuity is worse than the extractive abstract, and finally the generated abstract effect is not good.

[0004] However, in the field of Chinese abstract generation, the existing text abstract method still has many defects. Although the current method can basically summarize the main content of the original text, there are still some deficiencies in many details. The most common problem is the loss of key information. Simply copying information from the original text during abstract generation can easily lead to the loss of key information. In addition, the out-of-vocabulary generation word problem is still very common. A good abstract generation model should be abstract and even reflect the high-level semantic information of the original text content. The existing model still has room for improvement in these aspects.

[0005] TECHNICAL SCHEME

[0006] The application provides a text abstract generation method based on keyword information and a multi-head attention mechanism, and solves technical problems such as unclear semantic expression and keyword loss in the abstract generation process.

[0007] The application can be implemented through the following technical solutions:

[0008] A text abstract generation method based on keyword information and a multi-head attention mechanism comprises the following steps:

[0009] Step one, constructing an abstract generation model

[0010] The classical seq2seq network model with an integrated attention mechanism is used as the main body, and a keyword acquisition branch is added, which is used for keyword extraction and filtering on the input text to obtain refined keywords, and the refined keywords are input into the attention mechanism module of the classical seq2seq network model for subsequent abstract generation.

[0011] Step two, collecting Chinese texts, constructing a data set, training the abstract generation model, and then using the trained abstract generation model to automatically generate abstracts for the to-be-inspected texts;

[0012] The classical seq2seq network model with an integrated attention mechanism is used to process the original text input, and the original text input is represented by x={x1, x2, x3…x i}, wherein x i represents the i-th word in the original text, after word embedding processing, it enters the model through Bi-LSTM and becomes an encoder hidden layer state h i one by one, and the final output is represented by y={y1, y2, y3…y m}, wherein y m represents the m-th word of the output abstract.

[0013]

[0014] wherein, and respectively represent the forward state and the reverse state of the word in the LSTM hidden layer, represents the encoder hidden layer state.

[0015] Then, through the distribution of the attention weight , and the multiplication of the attention weight , the context vector is finally obtained, and the attention weight is obtained from the hidden layer state , and the decoding layer state co-decide, wherein , , , are trainable parameters;

[0016]

[0017] Finally, the context vector is concatenated with the decoding layer state and goes through two linear layers and a softmax layer to get the predicted word distribution probability of the final summary denoted as P(w);

[0018]

[0019]

[0020] wherein, , , , are trainable parameters, denotes the word dictionary probability distribution of the final generated summary.

[0021] Further, the keyword obtaining branch includes a keyword extraction module and a filtering module, the keyword extraction module extracts keywords by using a keybert extraction technology, and the filtering module uses a multi-head self-attention mechanism based on a transformer structure to perform weight redistribution on the extracted keywords, so as to adjust the word vector size of each keyword and highlight the keyword vector K containing more key information.

[0022] Further, the calculation formula of the multi-head self-attention mechanism is as follows:

[0023]

[0024] wherein, denotes the output of the i-th attention in the multi-head self-attention mechanism, is a corresponding linear matrix, the outputs of the attentions are spliced, and then linear operation is performed to obtain the keyword vector K after weight distribution.

[0025] Further, the loss function for the summary generation model is set as wherein, denotes a hybrid learning training loss function, is a reinforcement learning loss function, represents a maximum likelihood estimation loss function, and γ represents a hyperparameter between 0 and 1.

[0026] Further, the input text is segmented using the jieba segmenter, and then useless labels, special symbols and stop words are removed, and then the excessively long input text is truncated, the maximum length is set to 100, the input text preprocessing is completed, and finally the preprocessed data set is divided into training set, validation set and test set.

[0027] Further, the pointer mechanism is introduced into the decoder of the classical seq2seq network model for generating the abstract.

[0028] The beneficial technical effects of the present application are as follows:

[0029] 1. The keybert keyword extractor is used to extract the key information in the original text content, the appropriate number of keywords is selected according to the length characteristics of the Chinese abstract, and the multi-head self-attention mechanism is added to filter the keyword information, so that the keyword information is given appropriate weight, and the model has better context information fusion ability and key information expression ability.

[0030] 2. The decoder with the fusion pointer mechanism is used, and a hybrid training strategy based on reinforcement learning is added, the created word table and the extracted words in the original text are combined to form the generated abstract, and the common out-of-vocabulary generation word problem and exposure bias problem in the text abstract are solved.

[0031] The experimental results show that on the NLPCC2017 data set, the model proposed in the present application has a certain degree of improvement in the ROUGE scoring indicators of the comparative model, ROUGE-1, ROUGE-2 and ROUGE-L, and according to the generated examples, the generated abstract effectively fuses the keyword information and has good readability. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 It is a structure diagram of the classical seq2seq network model of the present application;

[0033] Figure 2 It is a structure diagram of the abstract generation model of the present application;

[0034] Figure 3 It is a structure diagram of the keyword extraction branch of the present application. DETAILED DESCRIPTION

[0035] The specific embodiments of the present application will be described in detail below in combination with the drawings and preferred embodiments.

[0036] As Figure 2As shown, the application provides a text summary generation method based on keyword information and multi-head attention mechanism, taking a classical seq2seq network model with attention mechanism as the main body, adding a keyword acquisition branch, the keyword acquisition branch is used for keyword extraction and filtering of the input text to obtain refined keywords, and then the refined keywords are input into the attention mechanism module of the classical seq2seq network model for subsequent summary generation, so as to construct a summary generation model; collect Chinese text, construct a data set, train the summary generation model, and then use the trained summary generation model to generate automatic summary of the text to be detected, wherein the keyword acquisition branch includes a keyword extraction module and a filtering module, the keyword extraction module uses keybert extraction technology for keyword extraction, and the filtering module uses a multi-head self-attention mechanism based on a transformer structure to re-allocate the weight of the extracted keywords, thereby adjusting the word vector size of each keyword, and highlighting the keyword vector K containing more key information.

[0037] In this way, the keyword information is acquired by means of the keyword extraction module, the keyword information is filtered by introducing the multi-head self-attention mechanism, the perception of the key information in the original text is strengthened, the semantic level of the summary is enriched, and a certain filtering and fusion are performed, the details of the key information are improved, and the problems such as exposure bias in generating the summary are solved. Experimental results show that the model has a significant effect on improving the effect of text summary, and the generated summary effect example also shows that the model has good readability and pays attention to the keyword information of the original text.

[0038] Specifically as follows:

[0039] S1, acquire a data set, the data set is a public Chinese text summary data set, in this embodiment, the data set used is the NLPCC2017 text summary data set,

[0040] S2, text preprocessing, preprocessing the acquired data set text, and dividing the preprocessed text into training set and test set;

[0041] jieba tokenizer is used for word segmentation processing of the text;

[0042] Then remove useless labels, special symbols and stop words;

[0043] Finally, the excessively long original text is truncated, and the maximum length is set to 100;

[0044] The preprocessed data set is divided according to the proportion, wherein the training set: validation set: test set is 98:1:1.

[0045] S3, construct an abstract generation model based on keyword information and a multi-head self-attention mechanism, train the model according to the obtained training set, and then detect the to-be-detected text by using the trained model;

[0046] The sequence-to-sequence seq2seq model fused with the attention mechanism is a classical seq2seq model which can be described as an encoder-decoder module, which is composed of an encoder and a decoder, and the traditional model encoder and decoder are composed of RNN, and the LSTM based on RNN gradually replaces RNN to become mainstream. The present application takes the classical seq2seq model as the main model, uses a single-layer bidirectional LSTM as the encoder, and adopts a common unidirectional LSTM structure for the decoder. The model structure is as shown in Figure 1 .

[0047] The input of the original text is represented by x={x1, x2, x3…xi}, wherein xi represents the i-th word in the original text, after word embedding processing, through Bi-LSTM into the model and becomes an encoder hidden layer state hi one by one, then the context vector ht* is obtained by connecting the bidirectional hidden layer state. The hidden layer state obtained in this way can better express the global information, and finally the output is represented by y={y1, y2, y3…ym}, wherein ym represents the m-th word of the output abstract.

[0048]

[0049] wherein, and respectively represent the forward state and the reverse state of the LSTM hidden layer of the word, represent the encoder hidden layer state.

[0050]

[0051] After encoding by Bi-LSTM, the hidden layer state of each word is obtained, then the attention weight is distributed, and finally the context vector is obtained by multiplying the attention weight. The attention weight is determined by the hidden layer state , the decoding layer state , wherein , , , are all trainable parameters.

[0052] The context vector After that, it is combined with the decoded layer state The connection is through two linear layers and a softmax layer to obtain the final vocabulary distribution.

[0053]

[0054] where, , , , are trainable parameters, The final generated summary word dictionary probability distribution is represented as P(w), and the predicted word distribution probability of the final summary is represented as P(w):

[0055]

[0056] In order to enhance the semantic features of the input information and the perception ability of the key information, the application adds a key acquisition branch on the basis of the main model, and the specific structure is as shown in Figure 2 The key acquisition module is as shown in Figure 3 The key acquisition module proposed by the application is divided into two parts, an extraction part and a filtering part, wherein the extraction part is composed of a keybert keyword extractor, and the filtering part is composed of a multi-head self-attention mechanism module based on a transformer structure.

[0057] Keybert is a simple and powerful method, which was proposed by Sharma in 2019, unlike traditional statistical-based LDA and TF-IDF keyword extraction algorithms, keybert first uses BERT to obtain the vector-level representation of the original document, so that it contains rich semantic information, then calculates the cosine similarity between n-gram or phrase and the original document, and the cosine similarity between the source document vector y and the i-th n-gram-based word vector is as follows: , The cosine similarity calculation formula is as follows:

[0058]

[0059] According to the results, find the phrase or phrase in the document with the highest similarity to the original text, and extract it as key information. The application selects three key words in the original text as key information by using keybert in combination with Chinese writing habits and the length of the generated summary, and uses the word embedding tool of keybert for vectorization.

[0060] Since the importance of different keywords in the original text is different, in order to filter the keyword information, a keyword filtering mechanism is proposed in the paper. Considering that the multi-head self-attention mechanism based on Transformer has better results in short text performance, the multi-head self-attention mechanism is selected as the keyword filter. After obtaining the keyword information output by keybert, it is input into the multi-head attention mechanism. The three initialization matrices Q, K and V are randomly initialized according to the keyword information. After a series of operations after Q, KT multiplication, the attention weight is obtained, and the attention weight is calculated by using As the denominator, d represents the dimension of the input matrix Q, K. After matrix multiplication and scaling, the attention weight is obtained through the softmax layer and the dot product with the matrix V. The formula for calculating the weight Attention (Q, K, V) is as follows:

[0061]

[0062] In the multi-head attention mechanism, multiple attention layers are usually processed in parallel and the final output is obtained by connection, and the formula for calculating the single attention mechanism and the total result is as follows:

[0063]

[0064]

[0065] Among them, represents the output of the i-th attention in the multi-head self-attention mechanism, is the corresponding linear matrix, and the outputs of each attention are spliced and then linearly operated to obtain the keyword vector K after weight distribution.

[0066] The model of the application extracts a fixed length word sequence for a given text content, inputs the Bi-LSTM encoding layer through the word embedding module, and obtains the hidden layer state. Let the hidden layer state of the i-th encoder unit be , the decoder hidden layer state at time t is , and the keyword vector obtained after the keybert keyword extractor and the multi-head self-attention mechanism is K.

[0067] The encoder hidden layer state , the decoder hidden layer state , and the keyword vector K jointly determine the attention weight, and the calculation formula of the attention weight is as follows:

[0068]

[0069] After obtaining the attention weight ​Afterwards, in order to limit the repetitiveness of the generated words, the present application follows the way of Palus to punish the words with higher weights, and then carries out a layer of softmax normalization, and then combines with the hidden layer state The context vector is calculated The calculation formula is as follows:

[0070]

[0071]

[0072]

[0073] Among them, represents the attention weight, represents the encoder hidden layer state, represents the context vector.

[0074] In order to effectively prevent the OOV problem, the present application introduces a pointer mechanism for decoding to generate the summary. This structure allows the generated summary to be composed of words extracted from the original text and the word table at the same time, represents the probability of obtaining a word from the word table for generating the summary, and the calculation method is known from the above. In the decoder of the present application, the calculation of the attention weight and the context vector is shown in the above formula, represents the probability of extracting a word from the word table, which is calculated by , , together, The calculation formula is as follows:

[0075]

[0076] Among them, , , , are trainable parameters, and σ represents the sigmoid function for normalizing the numerical value. After obtaining the probability of generating a word from the word table , it is used in the formula for finally generating the word probability, which is used to obtain the words in the word table and the words out of the word table at the same time, and the generation probability of the words out of the word table is (1- ), and the total probability distribution formula is calculated as follows:

[0077]

[0078] P(w) represents the word probability distribution of the summary generation, represents the probability distribution of generating a word from the word table, if it is 0, it represents that all the generated summary words are obtained from the original text; on the contrary, When the value is 0, it indicates that all the generated summary words are obtained from the word table.

[0079] In the traditional summary generation process, maximum likelihood estimation is mostly used as the loss function and good results are achieved. In general sequence-to-sequence generation tasks, the maximum likelihood estimation (MLE) algorithm is widely used. MLE is obtained by maximizing the probability of the given sample in the sequence-to-sequence training process, and the original text of an article is defined as x, and the generated summary is y={y1*, y2*, y3*…}, the best result is obtained by minimizing the following loss function in training:

[0080]

[0081] wherein, represents the loss function based on MLE. The method based on MLE can generate words one by one through comparison with the standard summary in the training process, but there is no such supervision in the test, and this problem is called exposure bias problem; at the same time, this method is not combined with the final generation summary evaluation method ROUGE, and many potential effective summaries are ignored. Therefore, the model adds reinforcement learning to solve this problem. Reinforcement learning is an algorithm that optimizes the evaluation index to improve the accuracy of the model. The reinforcement learning method added in the model follows the training method of Paulus. For a given input sequence, in each time step, represents the probability distribution of each decoder, the maximum word in the probability distribution output is obtained by greedy selection, is obtained by random selection in the probability distribution output, and represents the reward function of the output sequence y, and is compared with the standard output sequence by the ROUGE evaluation index. It is realized by minimizing the reinforcement loss, and the calculation method is as follows:

[0082]

[0083] The application uses a hybrid learning strategy, combines maximum likelihood estimation and reinforcement learning for training, reduces the exposure bias problem while maintaining sequence readability. The calculation formula is as follows:

[0084]

[0085] wherein, represents the hybrid learning training loss function, is the reinforcement learning loss function, where represents the maximum likelihood estimation loss function, and represents a hyper-parameter between 0 and 1.

[0086] S4, experimental verification

[0087] The present application carries out experiments on the NLPCC2017 text summary dataset, which is a dataset provided by the CCF International Natural Language Processing and Chinese Computing Conference in 2017 single document news summary evaluation task, and contains a total of 50000 news data samples. According to statistics, the average length of the original text in the dataset is 990, and the average length of the summary is 44. In this experiment, 49500 samples are selected as the training set and the verification set, and the remaining 500 samples are selected as the test set.

[0088] The model in this paper sets the maximum length of the input text during training to 100, the maximum length of the generated summary to 20, the word embedding dimension to 256, the hidden state dimension to 512, the word table size to 50000, the batch size to 36, the initial learning rate to 1e-03, the learning rate of RL to 1e-04, and the optimizer to the adam optimizer.

[0089] The experiment uses ROUGE as the evaluation index, which is the most common evaluation index in text summary. This index evaluates the repetition rate of the model generated summary and the standard summary (artificially generated summary) based on n-gram. This method is widely used in text summary, and the calculation method of ROUGE includes accuracy P, recall R and F value. Since F value considers both accuracy and recall in the calculation process, this paper uses the F value of ROUGE-1, ROUGE-2 and ROUGE-L to evaluate and calculate the comparison of the three evaluation indexes. The calculation method of ROUGE-N and ROUGE F value is as follows:

[0090]

[0091]

[0092] where RS represents the reference summary, represents the maximum number of n-grams that are the same in the model generated summary and the reference summary, represents the number of n-grams in the reference summary, P and R represent the accuracy and recall of ROUGE, respectively.

[0093] The comparison experiment results of the model of the present application and the existing model are shown in Table 1, and according to the results in Table 1, it can be seen that:

[0094] (1) The PGN-based model has a greater improvement in overall accuracy than the traditional seq2seq+Attention model, which shows that the PGN-based model still has good performance.

[0095] (2) After adding the keybert keyword extraction module, the accuracy of the generated summary is improved to a certain extent, but the improvement is not great. This shows that the keyword information has little effect on the accuracy of the model after being input without processing.

[0096] (3) After adding the reinforcement learning mechanism, the model effect is obviously improved, and the PGN-based model is improved by 1.55% and 0.6% in ROUGE-1 and ROUGE-L, respectively, which shows that the reinforcement learning mechanism has a significant effect on the score improvement and has a great contribution to the accuracy and fluency of the generated summary.

[0097] (4) The addition of multi-head self-attention makes the model achieve the best results in ROUGE-1 and ROUGE-L, with an improvement of 2.23% and 1.72%, respectively, and only ROUGE-2 is slightly behind. This shows that simply adding keyword information is not enough to make the key information well integrated into the model, and the filtering effect of multi-head self-attention is needed to allocate the weight of the keyword.

[0098] (5) The model of the present invention achieves the best results in ROUGE-1 and ROUGE-L, with a 0.17% lag behind the PGN model in ROUGE-2. This shows that the model of the present invention has good results in the accuracy and fluency of the generated summary.

[0099] Table 1 ROUGE F value score results of NLPCC2017 dataset

[0100]

[0101] The traditional seq2seq+Attention model is prone to the problem of generating words that are not registered, and the introduction of PGN can better improve this problem. The keybert keyword extraction module can help generate detailed summaries, but the keyword information may not be accurate enough, and some details are still not good enough. After adding the reinforcement learning method, the summary content is more concise and close to the standard summary content, which shows that reinforcement learning has a positive effect on the improvement of the accuracy of the generated summary.

[0102] The model of the present invention introduces a multi-head self-attention mechanism to filter keyword information based on the above methods, enriches the semantic level of the summary, and improves the details of the key information, which shows the effectiveness of the model of the present invention.

[0103] In summary, the application proposes a new text summary model based on keyword information and pointer generation network for improving the traditional pointer generation network, and generates a summary that fuses keybert keyword information and multi-head self-attention mechanism. A new hybrid learning strategy is also used for training, which enriches the semantic level of the generated summary, maintains the readability of the output sequence, strengthens the perception of key information in the original text, and performs certain filtering and fusion, and also solves the exposure bias problem of the generated summary. Experimental results show that the model has a significant effect on the improvement of the text summary effect, and the generated summary effect example also shows that the model has good readability and pays attention to the key information of the original text.

[0104] Although the specific embodiments of the application are described above, those skilled in the art should understand that these are only illustrative, and various changes or modifications can be made to these embodiments without departing from the principles and essence of the application, therefore, the protection scope of the application is defined by the appended claims.

Claims

1.A method for text summarization based on keyword information and multi-head attention mechanism, characterized in that Comprise the following steps: Step one, build abstract generation model The classical seq2seq network model with attention mechanism is used as the main body, and a keyword acquisition branch is added, which is used for keyword extraction and filtering of the input text to obtain refined keywords, and then the refined keywords are input into the attention mechanism module of the classical seq2seq network model for subsequent abstract generation; Step two, collect Chinese text, build dataset, train abstract generation model, and then use the trained abstract generation model to generate automatic abstract for the text to be detected; The classical seq2seq network model with attention mechanism is adopted to process the original text input represented by x={x1, x2, x3...x i}, wherein x i represents the i-th word in the original text. After word embedding processing, the model is entered through Bi-LSTM and becomes an encoder hidden layer state h i one by one. Finally, the output is represented by y={y1, y2, y3...y m}, wherein y m represents the m-th word of the output abstract. wherein, and denote the forward and backward state representations of the word in the LSTM hidden layer, respectively, denote the encoder hidden layer states; Subsequently, the context vector is obtained by multiplying the attention weight with the hidden layer state and the decoding layer state , where the attention weight is determined by the hidden layer state and the decoding layer state , , , are trainable parameters. Finally, the context vector is concatenated with the decoding layer state and passed through two linear layers followed by a softmax layer to obtain the predicted word distribution probability for the final summary, denoted by P(w). where, , , , are trainable parameters, denotes the word dictionary probability distribution of the final generated summary. 2.The method of claim 1, wherein: The keyword acquisition branch includes a keyword extraction module and a filtering module, the keyword extraction module uses keybert extraction technology for keyword extraction, and the filtering module uses a multi-head self-attention mechanism based on a transformer structure to redistribute the weights of the extracted keywords, thereby adjusting the word vector size of each keyword and highlighting the keyword vector K containing more key information. 3.The method of claim 2, wherein: The calculation formula of the multi-head self-attention mechanism is as follows: denotes the output of the i-th attention in the multi-head self-attention mechanism, The outputs of each attention are spliced for the corresponding linear matrix, and then linearly operated to obtain the keyword vector K after weight distribution. 4.The method of claim 1, wherein: The loss function for the abstract generation model is set as wherein, represents a hybrid learning training loss function, is a reinforcement learning loss function, represents a maximum likelihood estimation loss function, and γ represents a hyperparameter between 0 and 1. 5.The method of claim 1, wherein: Use jieba tokenizer to perform word segmentation on the input text, then remove useless labels, special symbols and stop words, then truncate the input text if it is too long, set the maximum length to 100, complete the input text preprocessing, and finally divide the preprocessed dataset into training set, validation set and test set. 6.The method of claim 1, wherein: The decoder of the classical seq2seq network model introduces a pointer mechanism to generate abstracts.

Images