Generative text summarization method based on key information masking and copying

Abstract

The present invention provides a generative text summarization method based on key information masking and copying. The method comprises the following steps: obtaining an original text sequence of an original document in units of words to obtain a key information set; obtaining an input text sequence; constructing a key information masking language model, and obtaining a masking matrix after the input text sequence passes through the key information masking language model to obtain a training set; constructing a generative text summary model, comprising an encoder, a decoder and a BIOK copy mechanism for fusing key information to generate a text summary; obtaining a text summary of the input document through the trained generative text summary model; and using the key information-based masking language model and the BIOK copy mechanism, the present invention enables the model to acquire the ability to recognize and summarize key information, thereby reducing the number of out-of-set words in the generated text and effectively improving the generation quality of the text summary.

Description

Technical Field

[0001] The present invention relates to a generative text summarization method based on key information masking and copying, and belongs to the technical field of natural language processing. Background Art

[0002] The generative text summarization task aims to understand text content through model training, mimicking human thinking to compress and restructure the original text to generate a summary. Compared to extractive methods, generative methods do not require evaluating the importance of the text, and the words or sentences in the generated summary do not need to be completely from the original document. As a result, the generated summary has higher linguistic coherence and lower semantic redundancy.

[0003] However, generative methods are text generation tasks that involve underlying language processing methods such as sentence compression, paraphrasing, and synonym replacement, which undoubtedly increases the difficulty of model design and training. In the early days of text summarization research, extractive methods developed rapidly. However, with the emergence of large-scale text summarization datasets and the rise of deep learning technology, people gradually turned their attention to generative text summarization tasks.

[0004] In recent years, although sequence-to-sequence models integrating attention mechanisms have been widely used in text generation tasks and have achieved significant results in machine translation tasks, their performance in generative text summarization tasks still needs to be improved.

[0005] Currently, generative methods face two major challenges: First, most methods only model the original document, while summaries typically consist of key information from the document. These methods ignore the impact of this key information on the document's main theme. Second, these methods often use copying mechanisms to mitigate the out-of-text problem, but traditional copying mechanisms are unable to copy key and continuous information from a text sequence. This lack of modeling of key information in a text sequence results in poor quality summaries.

[0006] The above problems are issues that should be considered and solved in the process of text summary generation. Summary of the Invention

[0007] The purpose of the present invention is to provide a generative text summarization method based on masking and copying key information to solve the problems in the prior art of insufficient attention to key information and the need to improve the quality of generated summaries.

[0008] The technical solution of the present invention is:

[0009] A generative text summarization method based on key information masking and copying includes the following steps:

[0010] S1. Obtain an original text sequence of an original document in units of words, use part-of-speech tagging technology and information extraction technology to obtain a noun set and a keyword set respectively, and calculate their intersection to obtain a key information set;

[0011] S2. Obtain a text sequence containing key information from the original text sequence and the key information set, and combine it with the given target sequence to obtain an input text sequence;

[0012] S3. Build a key information mask language model, input the text sequence through the key information mask language model, obtain the mask matrix, and obtain the training set;

[0013] S4. Construct a generative text summary model, including an encoder, a decoder, and a BIOK copy mechanism that integrates key information. The encoder converts the input text into the corresponding vector representation, and the decoder converts the encoder's L-th layer output h L As input to the decoder, a word list unit probability distribution and a reconstructed copy tag sequence are generated. The BIOK copy mechanism that integrates key information is used to optimize the word list probability distribution of the generated text. The optimized word list unit probability distribution is combined with the beam search algorithm to generate a text summary.

[0014] S5. The generative text summary model constructed in step S4 is trained using the training set obtained in step S3 to obtain a trained generative text summary model, and a text summary of the input document is obtained through the trained generative text summary model.

[0015] Furthermore, in step S1, the original document is subjected to part-of-speech tagging technology and information extraction technology to extract the noun set and keyword set in the document respectively, and their intersection is calculated to obtain the key information set, specifically,

[0016] S11. For the original document, use word segmentation technology to represent the document as an original text sequence D = (w1, w2, ... w p ), where p is the number of words in the text sequence D;

[0017] S12. For word w in the text sequence i , use part-of-speech tagging technology to tag the word w i The part of speech is f i , filter out the noun parts of speech in the text and form a noun set D n =(w n1 ,w n2 ,…,w no ), where o is the number of nouns; the keyword set D in the text is extracted using the graph-based sorting algorithm, namely the TextRank algorithm k =(k1,k2,…,k s), where s is the number of keywords;

[0018] S13, keyword set D k and noun set D n The intersection of the original text sequence D is called the key information set KW D =(kw1,kw2,…,kw g ), where g is the number of key information.

[0019] Furthermore, in step S3, the key information mask language model uses a masking operation on the target sequence in the input text sequence, and performs an alternating dynamic masking operation on the key information in the input text sequence.

[0020] Furthermore, in step S2, the text sequence containing key information is obtained from the original text sequence and the key information set, and the input text sequence is obtained by combining the given target sequence, specifically,

[0021] S21, based on the original text sequence and the key information set, replacing words in the original text sequence with key information in the corresponding key information set to obtain a text sequence containing key information;

[0022] S22. After concatenating the text sequence containing the key information with the given target sequence, an input text sequence is obtained.

[0023] Furthermore, in step S4, the encoder stacks L layers of encoding calculation units in total, and the encoding process of the encoder layer l is:

[0024] In the multi-head attention mechanism layer of the encoder layer l, the output h of the l-1 layer is l-1 As the input of the current layer, the mask matrix M constructed using the key information mask language model k Combined with the self-attention mechanism, the output result a of the current layer of the multi-head attention mechanism layer is calculated l :

[0025] Q=h l-1 W l Q , K=h l-1 W l K , V=h l-1 W l V

[0026]

[0027] Among them, Q, K, V are random matrices, softmax is the normalized exponential function, W l Q ,W lK ,W l V is the trainable parameter matrix, d k is the dimension of matrix K; M k is the mask matrix: Mask matrix M k Each element in has only two states: 0 and -∞. -∞ represents the masked information, and 0 represents the normal and valid information.

[0028] u l =LayerNorm(h l-1 +a l )

[0029] h l =LayerNorm(u l +FFN(u l ))

[0030] Among them, h l-1 Represents the hidden layer variable output by the encoder at layer l-1, a l is the output result of the current layer of the multi-head attention mechanism layer, u l is the intermediate result, LayerNorm and FFN are the normalization layer and feedforward neural network layer respectively.

[0031] Furthermore, in step S4, the decoder converts the Lth layer output h of the encoder L As the input of the decoder, the probability distribution of vocabulary units and the reconstructed copy label sequence are generated, specifically,

[0032] S41, through the gated unit recurrent network layer GRU and the normalization layer, the hidden layer variable h at the decoding end is obtained d , the calculation process is shown in the following formula:

[0033] r d =GRU(h L )

[0034] h d =LayerNorm(r d )

[0035] Among them, r d is the intermediate variable passing through GRU, and LayerNorm is the normalization layer;

[0036] S42, for the hidden variable h d After two different linear changes through the linear layer, the probability distribution P of the vocabulary unit is obtained by the Softmax function. vocab and the reconstructed copy tag sequence As the output of the decoder, the calculation process is as follows:

[0037] P vocab =softmax(W v h d +b v )

[0038]

[0039] Among them, W v , W e , b v , b e is a trainable model parameter, and softmax is a normalized exponential function.

[0040] Furthermore, in step S4, the BIOK copy mechanism that integrates key information is used to optimize the vocabulary probability distribution of the generated text, and the optimized vocabulary unit probability distribution is combined with the beam search algorithm to generate a text summary, specifically,

[0041] S43. For the input text sequence x input , the decoder of the generative text summarization model outputs the probability distribution of the vocabulary unit at the current time i and copy tags

[0042] S44, by copying the label Optimize the vocabulary probability distribution of generated text and obtain the optimized vocabulary unit probability distribution

[0043] S45, using the optimized vocabulary unit probability distribution Combined with the beam search algorithm, the text sequence at the current moment is generated to obtain the text summary.

[0044] Furthermore, in step S44, by copying the label Optimize the vocabulary probability distribution of generated text and obtain the optimized vocabulary unit probability distribution Specifically,

[0045] S441, if This means that the decoder's current generation unit will be copied from the original text sequence. Remove all unit distributions that are not in the original sequence to obtain the optimized vocabulary unit probability distribution

[0046] S442, if This means that the decoder's current generation unit and the previous generation unit both come from the original text sequence. Remove all unit distributions that cannot form a continuous sequence with the original text sequence to obtain the optimized vocabulary unit probability distribution

[0047] S443, if It indicates that the generation unit of the decoder at the current moment will come from the key information set, then Remove all unit distributions that cannot form a continuous sequence with key information to obtain the optimized vocabulary unit probability distribution

[0048] S444, if This means that the decoder's current generation unit will not be in the original sequence. Without making any adjustments, we can obtain the optimized probability distribution of vocabulary units.

[0049] The beneficial effects of the present invention are:

[0050] 1. This generative text summarization method based on key information masking and copying, by using a masked language model based on key information and a BIOK copy mechanism, will enable the model to acquire the ability to identify and summarize key information, reduce the number of out-of-character words in the generated text, and effectively improve the generation quality of text summaries. It can solve the problem of insufficient attention to key information and alleviate the generation of out-of-character words in current generative summary models.

[0051] Second, in this invention, under the action of the key information masking language model, not only does the model learn to generate summaries based on the original document, but also, because the key information in the original document is masked, the model's ability to capture key information in the text sequence is enhanced, thereby assisting in the generation of the final summary. The dynamic masking training method for key information improves the model's ability to identify key information.

[0052] 3. This generative text summarization method based on key information masking and copying optimizes the key information generation process by using the BIOK copy mechanism, enhances the model's attention to key information in the text, and enables the generated text to contain more key information in the original text.

[0053] Fourth, this generative text summarization method, based on masking and copying key information, employs a BIOK copying mechanism that incorporates key information, completing the sequence copying task with a single layer of a feedforward neural network. This reduces implementation and computational complexity and allows for copying specific fragments or key information from the original text, making the copied information more valuable. During the decoding phase, a gated unit neural network is used to enhance the contextual features of the text. This method also improves the fidelity and semantic coherence of the generated text to the original. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 4 is a flow chart of a generative text summarization method based on key information masking and copying according to an embodiment of the present invention.

[0055] Figure 2 The mask matrix M is obtained in the embodiment k Schematic diagram of the description.

[0056] Figure 3 Schematic diagram illustrating an encoder and a decoder in the embodiment.

[0057] Figure 4 Schematic diagram illustrating the BIOK copy mechanism integrating key information in the embodiment.

[0058] Figure 5 3. It is a schematic diagram comparing the experimental results of the out-of-word ratio of the KMCS model of the embodiment, the B-Unilm model, and the PointerNet model.

[0059] Figure 6 3 is a schematic diagram comparing experimental results of key information copying amount of the KMCS model and the B-Unilm model of the embodiment. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0061] Example

[0062] A generative text summarization method based on key information masking and copying, such as Figure 1 , including the following steps,

[0063] S1. Obtain the original text sequence of the original document in units of words, use part-of-speech tagging technology and information extraction technology to obtain a noun set and a keyword set respectively, and calculate their intersection to obtain a key information set.

[0064] S11. For the original document, use word segmentation technology to represent the document as an original text sequence D = (w1, w2, ... w p ), where p is the number of words in the text sequence D;

[0065] S12. For word w in the text sequence i , use part-of-speech tagging technology to tag the word w i The part of speech is f i , filter out the noun parts of speech in the text and form a noun set D n =(w n1 ,w n2 ,…,w no ), where o is the number of nouns; the keyword set D in the text is extracted using the graph-based sorting algorithm, namely the TextRank algorithmk =(k1,k2,…,k s ), where s is the number of keywords;

[0066] S13, keyword set D k and noun set D n The intersection of the text sequence D is called the key information set KW D =(kw1,kw2,…,kw g ), where g is the number of key information.

[0067] S2. Obtain a text sequence containing key information from the original text sequence and the key information set, and combine it with the given target sequence to obtain the input text sequence.

[0068] S21, based on the original text sequence and the key information set, replacing words in the original text sequence with key information in the corresponding key information set to obtain a text sequence containing key information;

[0069] S22. After concatenating the text sequence containing the key information with the given target sequence, an input text sequence is obtained.

[0070] Taking the original sequence S1 = (t1, t2, t3) as an example, step S1 extracts t1 and t3 as key information kt1 and kt3 in the sequence, then the text sequence containing key information is represented as S1` = (kt1, t2, kt3), and is concatenated with the target sequence S2 = (t4, t5) to construct the following input sequence: S input =(CLS,kt1,t2,kt3,SEP,t4,t5,SEP), where CLS and SEP represent the beginning and end of a sequence respectively.

[0071] S3. Build a key information mask language model, input the text sequence through the key information mask language model, obtain the mask matrix, and obtain the training set;

[0072] In step S3, the key information mask language model uses a masking operation on the target sequence in the input text sequence, and performs an alternating dynamic masking operation on the key information in the input text sequence.

[0073] By constructing a key information mask language model, the sequence-to-sequence model can enhance its attention to key information. input=(CLS,kt1,t2,kt3,SEP,t4,t5,SEP) as an example. In addition to using masking operations on the target sequence, the key information mask language model will also perform dynamic masking operations on the key information kt1 and kt3, that is, kt1 and kt3 are masked alternately in each round of training. For example, kt1 is masked in the i-th round of training, and kt3 is masked in the i+1-th round of training. The masked text sequence is obtained to form a mask matrix M k ,like Figure 2 .

[0074] In the text summarization task, the input source sequence is the original document, and the target sequence is the reference summary. Under the influence of the key information masked language model, the model not only learns the ability to generate summaries based on the original document, but also because the key information in the original document is masked, the model's ability to capture key information in the text sequence is enhanced, thereby assisting in the generation of the final summary.

[0075] S4. Construct a generative text summary model, including an encoder, a decoder, and a BIOK copy mechanism that integrates key information. The encoder converts the input text into the corresponding vector representation, and the decoder converts the encoder's L-th layer output h L As the input of the decoder, generate the vocabulary unit probability distribution P vocab and the reconstructed copy tag sequence like Figure 3 ; Using the BIOK copy mechanism that integrates key information, the word list probability distribution of the generated text is optimized, and the optimized word list unit probability distribution is combined with the beam search algorithm to generate a text summary, such as Figure 4 ;

[0076] In step S4, the encoder uses the encoder architecture of the transformer model, that is, the encoder architecture of the transformer, with a total of L layers of encoding calculation units stacked. For the input text x input , the encoding process of the encoder layer l:

[0077] a l =Attention′(h l-1 )

[0078] u l =LayerNorm(h l-1 +a l )

[0079] h l =LayerNorm(u l +FFN(u l ))

[0080] Among them, h l-1Represents the hidden layer variable output by the encoder at layer l-1, a l is the output result of the multi-head attention mechanism layer, u l is the intermediate result, LayerNorm and FFN are the normalization layer and feedforward neural network layer respectively.

[0081] In step S4, in the multi-head attention mechanism layer of the encoder layer l, the output h of the l-1 layer is l-1 As the input of the current layer, the mask matrix M constructed using the key information mask language model k Combined with the self-attention mechanism, the result a of the current layer is calculated l :

[0082] Q=h l-1 W l Q , K=h l-1 W l K , V=h l-1 W l V

[0083]

[0084] Among them, Q, K, V are random matrices, softmax is the normalized exponential function, W l Q , W l K , W l V is the trainable parameter matrix, d k is the dimension of matrix K; M k is the mask matrix: Mask matrix M k Each element in has only two states: 0 and -∞. -∞ represents the masked information, and 0 represents the normal and valid information.

[0085] In step S4, the decoder converts the Lth layer output h of the encoder L As the input of the decoder, generate the vocabulary unit probability distribution P vocab and the reconstructed copy tag sequence Specifically,

[0086] S41, through the gated unit recurrent network layer GRU and the normalization layer, the hidden layer variable h at the decoding end is obtained d , the calculation process is shown in the following formula:

[0087] r d =GRU(h L )

[0088] hd =LayerNorm(r d )

[0089] Among them, r d is the intermediate variable passing through GRU, and LayerNorm is the normalization layer;

[0090] like Figure 3 In the decoder, adding a gated unit recurrent network layer (GRU) and a normalization layer before the Softmax function can further capture the contextual features of the text sequence.

[0091] S42, for the hidden variable h d After two different linear changes through the linear layer, the probability distribution P of the vocabulary unit is obtained by the Softmax function. vocab and the reconstructed copy tag sequence As the output of the decoder, the calculation process is as follows:

[0092] P vocab =softmax(W v h d +b v )

[0093]

[0094] Among them, W v , W e , b v , b e is a trainable model parameter, and softmax is a normalized exponential function.

[0095] In step S4, the BIOK copy mechanism that integrates key information is used to optimize the vocabulary probability distribution of the generated text, and the optimized vocabulary unit probability distribution is combined with the beam search algorithm to generate a text summary.

[0096] S43. For the input text sequence x input , the decoder of the generative text summarization model outputs the probability distribution of the vocabulary unit at the current time i and copy tags

[0097] S44, by copying the label Optimize the vocabulary probability distribution of generated text and obtain the optimized vocabulary unit probability distribution

[0098] S441, if This means that the decoder's current generation unit will be copied from the original text sequence. Remove all unit distributions that are not in the original sequence to obtain the optimized vocabulary unit probability distribution

[0099] S442, if This means that the decoder's current generation unit and the previous generation unit both come from the original text sequence. Remove all unit distributions that cannot form a continuous sequence with the original text sequence to obtain the optimized vocabulary unit probability distribution

[0100] S443, if It indicates that the generation unit of the decoder at the current moment will come from the key information set, then Remove all unit distributions that cannot form a continuous sequence with key information to obtain the optimized vocabulary unit probability distribution

[0101] S444, if This means that the decoder's current generation unit will not be in the original sequence. Without making any adjustments, we can obtain the optimized probability distribution of vocabulary units.

[0102] S45, using the optimized vocabulary unit probability distribution Combined with the beam search algorithm to generate the current text sequence Get the text summary. The formal description of the above process is shown in Table 1:

[0103] Table 1 Text generation algorithm based on BIOK copy tags

[0104]

[0105] Where x represents the input text sequence x input The corresponding original text sequence, KW x Represents the input text sequence x input The corresponding key information set, MAX_L represents the maximum length of the generated sequence, count represents the number of iterations, v j Represents a text unit in the vocabulary, temp represents the copy sequence at the current moment, Represents the generation unit at the current moment; the function model represents the input function of the model and generates text according to the method of the unidirectional language model; the function beam_search represents the beam search algorithm; the function get_ngram is used to obtain the set of continuous sequences composed of text units in the copied sequence and the original sequence, and the result is stored in the array n_gram.

[0106] By adopting the BIOK copy mechanism that integrates key information, the BIO copy mechanism can improve its attention to key information in the text sequence and enhance its ability to adapt to text summarization tasks.

[0107] In the BIOK copy mechanism, B indicates that the unit is copied from the original sequence; I indicates that the unit is copied from the original sequence and forms a continuous segment with the previous unit; K indicates that the unit is copied from the original text and is key information; O indicates other characters. In the labeling process, it is stipulated that the longest common subsequence of the original sequence and the target sequence is the copy sequence c = (c1, c2, ..., c z ), z represents the length of the copy sequence. The first unit c1 of the copy sequence is marked as B; if the copy sequence contains key information of the original sequence and these units are not c1, the units containing key information are marked as K; the other text units of the copy sequence are marked as I. Figure 4 , take the original sequence S1 = (t1, t2, t3, t4) and the target sequence S2 = (t5, t2, t3, t4) as an example to illustrate, using the key information extraction of step S1 and the sequence-to-sequence masked language model of step S2, extract t3 as the key information kt3 in the sequence, and splice the sequence into: S input =(CLS,t1,t2,kt3,t4,SEP,t5,t2,kt3,t4,SEP), where CLS and SEP represent the beginning and end of a sequence respectively. The result is labeled S label :S label =(O,O,B,K,I,I,O,B,K,I,I), where the copy sequence c = (t2,kt3,t4,SEP). Therefore, in the target sequence, t2 is labeled as B, kt3 is labeled as K, the sequences (t4,SEP) and (t2,kt3) are consecutively labeled as I, and the remaining characters are labeled as O. To ensure consistency between the labels of the original and target sequences, the copy sequences contained in the original sequence are also labeled during actual model training.

[0108] S5. The generative text summary model constructed in step S4 is trained using the training set obtained in step S3 to obtain a trained generative text summary model, and a text summary of the input document is obtained through the trained generative text summary model.

[0109] Take the original text sequence x=(x1,x2,…,x n ), the reference summary is used as the target sequence y=(y1,y2,…,y m ) as an example to obtain the key information set KW x =(kw1,kw2,…,kw g), where n and m represent the number of words in the original text sequence and the number of text units in the reference summary, respectively, and g represents the number of key information in the original document. At the input of the model, the original text sequence and the reference summary are concatenated to obtain the sequence x input =(CLS,x,SEP,y,SEP). Use BIOK copy mechanism to mark sequence x input The label is b input =(b1,b2,…,b n+m+3 ).

[0110] The training goal of the generative text summary model in the training phase is to input and label b input Generate the text sequence and copy label sequence of the target sequence. The objective function is to train the parameters in the model by calculating the negative log-likelihood of the input and output:

[0111]

[0112] Among them, x input is the input text sequence, is the text sequence of the output target sequence, is the predicted copy label sequence, |D| is the number of training set samples, and θ is the model parameter.

[0113] This generative text summarization method based on masking and copying of key information, by using a masked language model based on key information and a BIOK copy mechanism, will enable the model to acquire the ability to identify and summarize key information, reduce the number of out-of-set words in the generated text, and effectively improve the generation quality of text summaries. It can solve the problem of insufficient attention to key information and alleviate the generation of out-of-set words in the current generative summary model.

[0114] In this invention, under the action of the key information mask language model, not only does the model learn to generate summaries based on the original document, but also, because the key information in the original document is masked, the model's ability to capture key information in the text sequence is enhanced, thereby assisting in the generation of the final summary. The dynamic masking training method for key information improves the model's ability to identify key information.

[0115] This generative text summarization method based on key information masking and copying optimizes the key information generation process by using the BIOK copy mechanism, enhances the model's attention to key information in the text, and enables the generated text to contain more key information in the original text.

[0116] This generative text summarization method, based on masking and copying key information, employs a BIOK copying mechanism that incorporates key information to complete the sequence copying task using only one layer of a feedforward neural network. This method reduces implementation and computational complexity and can copy specific fragments or key information from the original text, making the copied information more valuable. During the decoding phase, a gated unit neural network is used to enhance the contextual features of the text. This method also improves the fidelity and semantic coherence of the generated text to the original text.

[0117] This generative text summarization method based on key information masking and copying is based on the BERT+Seq2seq architecture. First, for an original document, it uses part-of-speech tagging and information extraction techniques to extract key information from the document. Secondly, it proposes a masked language model based on key information and a BIOK copy mechanism that incorporates key information. The former is used to adjust the input and training methods of the summary model, while the latter is used to optimize the vocabulary probability distribution of the generated text. Finally, the optimized vocabulary unit probability distribution is combined with a beam search algorithm to complete the summary generation.

[0118] This generative text summarization method based on key information masking and copying extracts key information from the text, extracts keywords and nouns from the document, and calculates their intersection to obtain a set of key information in the text, thereby enhancing the model's input. By building a key information masking model, a masking mechanism is used during the training process, and a mask matrix is ​​constructed to mask the padding information, thereby enhancing the model's ability to capture key information in the text sequence. By building the model's encoder and decoder, the encoder converts the input text into a corresponding vector representation using a corresponding mechanism, and the decoder converts the encoder's L-th layer output h L The decoder uses this as input to generate the corresponding sequence and label sequence. The BIO annotation model is improved, and a BIOK copy mechanism is constructed that incorporates key information. Furthermore, by incorporating this BIOK copy mechanism into the generated text, the vocabulary is optimized and the summary is generated. This BIOK copy mechanism, which incorporates key information, improves the BIO copy mechanism's focus on key information in the text sequence, enhancing its adaptability to text summarization tasks.

[0119] The generative text summarization method based on key information masking and copying in the embodiment is experimentally verified as follows:

[0120] In order to verify the effectiveness of the generative summary model based on key information masking and copying proposed in the embodiment, the current best models are selected for comparison: TextRank+BERT: This algorithm uses the BERT model to vectorize the sentences in the original document, and uses the cosine similarity between sentences to establish a full graph. The sentence with the highest score is selected as the document summary through algorithm iteration; RNN-Context: This model uses a recurrent neural network as an encoder and decoder, and takes words as units as input to complete the generation of summaries; NLP-ONE: This model uses a word model to encode the original sequence and the target sequence, and introduces an attention mechanism, and uses a decoder to complete the generation of summaries; SRB: This model uses a semantically associated neural network model, a gated unit encoder and a decoder with an attention mechanism to complete the generation of summaries; CopyNet: This model uses an encoder-decoder structure with an introduced copy mechanism, takes words as units as input, and completes the generation of text by calculating the copy probability and vocabulary probability. Summary generation; PointerNet: Based on the sequence-to-sequence model framework, this model selects summary words through a pointer network and uses a coverage mechanism to alleviate the problems of out-of-set words and information errors. The code used is from: https: / / github.com / yamonc / New-Pytorch-Chinese; AA: The model adopts an encoder-decoder architecture and completes summary generation by using a fusion attention mechanism to dynamically balance the local and global information of the original document; B-Unilm: This model adopts the BERT+Seq2seq architecture, uses a sequence-to-sequence language model as the input of the encoder, uses the pre-trained model BERT to complete text encoding, and uses the decoder to complete summary generation; W-Unilm: This model adopts the BERT+Seq2seq architecture, uses a sequence-to-sequence language model as the input of the encoder, uses the word-based pre-trained model WoBERT to complete text encoding, and uses the decoder to complete summary generation.

[0121] The model comparison and verification are as follows:

[0122] The experimental results of the KMCS model of the embodiment and each model on the LCSTS and NLPCC 2017 datasets are shown in Tables 1 and 2, respectively, where ROUGE-1, ROUGE-2, and ROUGE-L are ROUGE indicators, and R-Avg represents the average value of each ROUGE indicator.

[0123] Table 1 Comparison of evaluation indicators on the LCSTS dataset

[0124]

[0125] Table 2 Comparison of evaluation indicators on the NLPCC 2017 dataset

[0126]

[0127] The results in Table 1 and Table 2 show that, compared with other text summarization models in the two datasets, the KMCS model of the embodiment has improved ROUGE indicators. In various ROUGE indicators, the performance was improved by several percentage points compared with the RNN-Context model and the NLP-ONE model. Both models are constructed using recurrent neural networks and use the intermediate results of the encoder as the vector representation of the text, while the KMCS model of the embodiment uses a pre-trained model as the vector representation of the text. The improvement in evaluation indicators reflects the powerful language representation ability of the pre-trained model; in the LCSTS dataset, the performance was improved by 3.98%, 4.44%, and 1.9% respectively compared with the PointerNet model, and by 1.22%, 1.34%, and 1.53% respectively compared with the CopyNet model; in the NLPCC 2017 dataset, the performance was improved by 1.97%, 2.06%, and 2.28% respectively compared with the PointerNet model. Both the PointerNet and CopyNet models use a copy mechanism based on a pointer network, while the KMCS model of the embodiment uses a BIOK copy mechanism that is more suitable for Chinese text and the Transformer model. The improvement in evaluation indicators shows that the model of the embodiment enhances the ability to copy continuous and key information in the text, thereby improving the quality of summary generation.

[0128] In the LCSTS dataset, the R-AVG index is improved by 3.83% compared with the B-Unilm model. Both models use a method based on a sequence-to-sequence language model to complete the summary generation task. However, the KMCS model of the embodiment introduces a GRU network at the decoding end to enhance the sequence characteristics of the text. At the same time, it masks some key information during the training process, thereby improving the model's generalization ability and ability to capture key information. Compared with the W-Unilm model, the KMCS model of the embodiment has the characteristics of a wide vocabulary range, and uses a masked language model and copy mechanism based on key information to adjust the model's processing method for continuous word sequences, so that the model of the embodiment improves the R-AVG index by 1.22%. However, the word-based W-Unilm model reduces the number of iterations of summary generation (for example, the word-based model requires one step to generate a two-character word, while the character-based model requires two steps). Therefore, it is considered to use a word-based pre-training method in the future, and to perform pre-training from scratch in a hardware environment where conditions permit, to further improve the performance of the KMCS model.

[0129] The ablation experiment is as follows:

[0130] To verify the impact of the key information-based masked language model and copy mechanism on the performance of the KMCS model in this embodiment, these methods were removed and ablation experiments were conducted on the LCSTS dataset. The experimental results are shown in Table 3. The KMS model represents the KMCS model using only the key information masked language model, the KCS model represents the KMCS model using only the BIOK copy mechanism that incorporates key information, and the BIOS model replaces the BIOK copy mechanism in the KCS model with the BIO copy mechanism.

[0131] Table 3 KMCS model ablation experiment results of the embodiment

[0132] Model ROUGE-1 ROUGE-2 ROUGE-L KMCS 35.62 22.94 32.83 KCS 34.96 21.17 32.08 KMS 33.94 20.16 31.24 BIOS 34.01 20.98 31.87

[0133] In Table 3, the KCS model, which removes the key information masked language model, shows ROUGE performance decreases of 0.66%, 1.77%, and 0.75%, respectively, compared to the KMCS model of the embodiment. The KMS model, which removes the BIOK copy mechanism, shows a greater decrease in ROUGE performance than the KCS model, with ROUGE performance decreases of 1.68%, 2.78%, and 0.59%, respectively, compared to the KMCS model of the embodiment. These two sets of ablation experiment results show that the performance of the KMCS model of the embodiment degrades to varying degrees when removing the key information masked language model or the copy mechanism, and also verify that both approaches can improve the overall performance of the model. For the BIOS model, which uses the BIOK copy mechanism, ROUGE performance decreases of 0.95%, 0.19%, and 0.21%, respectively, compared to the KCS model. These ablation experiment results demonstrate that the BIOK copy mechanism, which improves the BIOK copy mechanism, helps improve the quality of summary generation.

[0134] The key information enhancement effect analysis is as follows:

[0135] To verify the impact of the masked language model and copying mechanism proposed in this example on summary quality, two indicators, the proportion of out-of-words and the amount of key information copied, are used.

[0136] The calculation of the proportion of out-of-words is as follows: Among them, PO represents the proportion of out-of-words, w represents the smallest unit of generated text, and w unk Indicates the generated UNK (out-of-set word) tag, Count(w) indicates the number of generated text units, Count(w unk ) represents the number of UNK tags generated. A smaller out-of-set word ratio (PO) indicates that the model generates fewer out-of-set words, indicating that the model is better at processing out-of-set words.

[0137] The calculation of the key information copy quantity is as follows: Among them, PK represents the amount of key information copied, kw represents the key information in the original text, and kw gen Indicates that key information kw appears in the generated text, kw gen The generation of PK indicates that the model is able to capture the key information in the original text when generating the summary. The larger the key information copy quantity PK, the more key information there is in the generated text, indicating that the model has a stronger ability to generate key information.

[0138] like Figure 5 The KMCS model of the embodiment and the PointerNet model using a copy mechanism both have lower out-of-text word ratios than the B-Unilm model, indicating that both models can reduce the number of out-of-text words in the generated text. Furthermore, compared to the PointerNet model based on a pointer network copy mechanism, the BIOK copy mechanism that integrates key information used by the KMCS model of the embodiment can complete the sequence copy task with only one layer of feedforward neural network. This reduces its implementation and computational complexity and allows it to copy specific fragments or key information from the original text, making the copied information more valuable.

[0139] like Figure 6 Compared to the B-Unilm model, the KMCS model of the embodiment increases the amount of key information copied by approximately 5%. Because the KMCS model of the embodiment uses a dynamic mask training method for key information during the encoding phase, the model's ability to identify key information is improved. Simultaneously, during the decoding phase, the gated unit neural network enhances the contextual features of the text and the BIOK copy mechanism optimizes the generation of key information. These methods enhance the model's focus on key information in the text, allowing the generated text to contain more key information from the original text.

[0140] The following is an example summary analysis:

[0141] Table 4 shows examples of summary generation using the KMCS model, RNN-Context model, and CopyNet model of the embodiment on the LCSTS test set.

[0142] Table 4 Summary generation examples of the KMCS model of the embodiment and other models on the LCSTS test set

[0143]

[0144] According to the results in Table 4, it can be seen that in the text generated by the RNN-Context model, only the meaning of "acquisition" is expressed, which is inconsistent with the meaning of "intended acquisition" expressed in the original text, and two UNK tags outside the vocabulary appear in the summary text; in the text generated by the CopyNet model, the first half of the sentence only expresses the meaning of "intended acquisition" in the original text, but does not clarify the object of acquisition, resulting in missing semantic information; in the text generated by the KMCS model of the embodiment, the key information "film and television" and "equity" are copied and generated, and the key information "company" is replaced with "Huayi Brothers" through model understanding. Except for the missing statement of the number of equity acquisitions, its generation result is consistent with the reference summary, indicating that the KMCS model of the embodiment can improve the fidelity and semantic coherence of the generated text to the original text.

[0145] The above experimental results show that this generative text summarization method based on key information masking and copying uses a masked language model based on key information and a BIOK copy mechanism, which enables the model to acquire the ability to identify and summarize key information, reduces the number of out-of-text words in the generated text, and improves the quality of summary generation.

[0146] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A generative text summarization method based on key information masking and copying, characterized by: The following steps are included: S1. Obtain an original text sequence of an original document in units of words, use part-of-speech tagging technology and information extraction technology to obtain a noun set and a keyword set respectively, and calculate their intersection to obtain a key information set; S2. Obtain a text sequence containing key information from the original text sequence and the key information set, and combine it with the given target sequence to obtain an input text sequence; S3. Build a key information mask language model, input the text sequence through the key information mask language model, obtain the mask matrix, and obtain the training set; S4. Construct a generative text summary model, including an encoder, a decoder, and a BIOK copy mechanism that integrates key information. The encoder converts the input text into the corresponding vector representation, and the decoder converts the encoder's first Layer output As input to the decoder, a word list unit probability distribution and a reconstructed copy tag sequence are generated. The BIOK copy mechanism that integrates key information is used to optimize the word list probability distribution of the generated text. The optimized word list unit probability distribution is combined with the beam search algorithm to generate a text summary. In step S4, the BIOK copy mechanism that integrates key information is used to optimize the word list probability distribution of the generated text, and the optimized word list unit probability distribution is combined with the beam search algorithm to generate a text summary, specifically, S43. For the input text sequence , the decoder output of the generative text summarization model is currently Probability distribution of vocabulary units at time and copy tags ; S44, by copying the label , optimize the vocabulary probability distribution of generated text and obtain the optimized vocabulary unit probability distribution ; S441, if , indicating that the decoder's current generation unit will be copied from the original text sequence, then Remove all unit distributions that are not in the original sequence to obtain the optimized vocabulary unit probability distribution ; S442, if , indicating that the decoder's current generation unit and the previous generation unit are both from the original text sequence, then Remove all unit distributions that cannot form a continuous sequence with the original text sequence to obtain the optimized vocabulary unit probability distribution ; S443, if , indicating that the generation unit of the decoder at the current moment will come from the key information set, then Remove all unit distributions that cannot form a continuous sequence with key information to obtain the optimized vocabulary unit probability distribution ; S444, if , indicating that the decoder's current generation unit will not be in the original sequence, then Without making any adjustments, we can obtain the optimized probability distribution of vocabulary units. ; S45, using the optimized vocabulary unit probability distribution Combined with the beam search algorithm to generate the current text sequence and obtain the text summary; S5. The generative text summary model constructed in step S4 is trained using the training set obtained in step S3 to obtain a trained generative text summary model, and a text summary of the input document is obtained through the trained generative text summary model.

2. The generative text summarization method based on key information masking and copying according to claim 1, characterized in that: In step S1, the original document is subjected to part-of-speech tagging technology and information extraction technology to extract the noun set and keyword set in the document respectively, and their intersection is calculated to obtain the key information set, specifically, S11. For the original document, use word segmentation technology to represent the document as a raw text sequence in units of words ,in, For text sequence The number of words; S12. For words in text sequences , using part-of-speech tagging technology to tag words The part of speech is , filter out the noun parts of speech in the text and form a noun set ,in, is the number of nouns; the graph-based sorting algorithm, TextRank algorithm, is used to extract the keyword set in the text ,in, is the number of keywords; S13. Gather keywords Set with nouns The intersection of Key information collection ,in, The number of key information.

3. The generative text summarization method based on key information masking and copying according to claim 1, characterized in that: In step S3, the key information mask language model uses a masking operation on the target sequence in the input text sequence, and performs an alternating dynamic masking operation on the key information in the input text sequence.

4. The generative text summarization method based on key information masking and copying according to claim 1, wherein: In step S2, the text sequence containing key information is obtained from the original text sequence and the key information set, and the input text sequence is obtained by combining the given target sequence, specifically, S21, based on the original text sequence and the key information set, replacing words in the original text sequence with key information in the corresponding key information set to obtain a text sequence containing key information; S22. After concatenating the text sequence containing the key information with the given target sequence, an input text sequence is obtained.

5. The generative text summarization method based on key information masking and copying according to claim 1, wherein: In step S4, the encoders are stacked Layer coding calculation unit, encoder Layer encoding process: In the encoder In the multi-head attention mechanism layer, the Output of the layer As the input of the current layer, the mask matrix constructed using the key information mask language model Combined with the self-attention mechanism, the output of the current layer of the multi-head attention mechanism layer is calculated : ， ， Among them, Q, K, and V are random matrices. is the normalized exponential function, is the trainable parameter matrix, is a matrix Dimensions; is the mask matrix: , the mask matrix Each element in has only 0 and Two states, Indicates the masked information, and 0 indicates normal and valid information; ， ， in, Indicates that the encoder is The hidden layer variables of the layer output, is the output result of the current layer of the multi-head attention mechanism layer, is the intermediate result, and They are the normalization layer and the feedforward neural network layer respectively.

6. The generative text summarization method based on key information masking and copying according to any one of claims 1 to 5, characterized in that: In step S4, the decoder converts the encoder's Layer output As the input of the decoder, the probability distribution of vocabulary units and the reconstructed copy label sequence are generated, specifically, S41, through the gated unit recurrent network layer GRU and normalization layer, obtain the hidden layer variables of the decoding end , the calculation process is shown in the following formula: ， ， in, is the intermediate variable after GRU, is the normalization layer; S42. For hidden variables After two different linear changes through the linear layer, the probability distribution of the vocabulary unit is obtained by the Softmax function. and the reconstructed copy tag sequence As the output of the decoder, the calculation process is as follows: ， ， in, are trainable model parameters, is the normalized exponential function.

Images