A text summarization method combining generative and extractive approaches

Abstract

The present invention discloses a method for generating text summarization that combines generative and extractive methods. The method includes the following steps: first, marking sentences in the original text as summaries on text summary data; then, during training, connecting a sentence encoding layer and a classification layer after the encoder of a generative pre-trained language model, and constructing a classification loss using the summary sentence markers to improve the ability to extract summary sentences; learning text representation through the encoder, and then learning sentence representation through the sentence encoding layer, splicing the text features output by the encoder and the sentence features output by the sentence encoding layer, and inputting them into a decoder to generate a summary of the original document. The sentence classification layer is not used during prediction. Compared with the existing technology, by introducing the sentence encoding layer and the classification layer between the encoder and the decoder of the pre-trained language model, the generative and extractive text summarization are integrated, and the sentences used as summaries are used to control and guide the generation process, thereby improving the quality of text summarization.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to the application of a generative pre-trained language model in a text summary generation scenario. Background Art

[0002] Text summarization is one of the text generation tasks. The core lies in the extraction, refinement and reorganization of summary information. It replaces the information of the entire text with a new paragraph. It is an information compression process. Traditional text summarization tasks are divided into extractive and generative. In practice, a pre-trained encoder is used for text classification to achieve the purpose of extracting important sentences in the text. A decoder is connected to the encoder to achieve the purpose of generating a summary. Therefore, it is natural to complete the extractive and generative text summarization tasks at the same time through an autoencoder decoder. Therefore, the present invention takes into account both extractive and generative text summarization at the same time through a generative pre-trained language model, and uses the extractive text summarization task to improve the generative text summarization effect.

[0003] Existing technologies are often limited to the fusion of loss functions and the separation of extraction and generation tasks. No technology has yet considered the establishment of a unified model for extraction followed by generation. In this practical scenario, the core issue lies in how to unify the optimization of extraction and generation. Summary of the Invention

[0004] The purpose of this invention is to provide a text summarization method that combines generative and extractive methods to solve the practical problem of information condensation in multi-style single text scenarios.

[0005] The present invention is designed to combine the characteristics of the pre-trained language model, use the encoder to encode the text, and the sentence encoding layer to extract important information to facilitate the decoder to generate a text summary that can replace the main idea of ​​the original document, and use dual optimization to solve the optimization problems brought about by different tasks.

[0006] The technical solution specifically adopted in the present invention is as follows:

[0007] A text summarization method that combines generative and extractive methods, the steps of which are as follows:

[0008] S1: Obtain the dataset required for text summarization training. Each sample in the dataset contains the original text and the summary text. Segment the original text and summary in each sample into sentences, and calculate the similarity between each sentence in the original text and each sentence in the summary text. Assign summary-related sentence labels to original text sentences with similarity above a threshold, so that each sample in the training set contains both the summary and the summary sentence label.

[0009] S2: Based on the generative pre-trained language model, a sentence encoding layer is cascaded between the encoder and decoder of the generative pre-trained language model to construct a text summarization model; in the text summarization model, sentences in the original text are input into the encoder, the encoder generates text representations, and then input into the sentence encoding layer to generate sentence representations. The text representations generated by the encoder and the sentence representations generated by the sentence encoding layer are fused and input into the decoder, which generates a summary corresponding to the original text;

[0010] S3. Adding a classification layer for auxiliary training to the sentence encoding layer in the text summarization model, wherein the sentence encoding layer and the classification layer constitute a classifier, classifying each sentence in the original text based on the sentence representation generated by the sentence encoding layer, and outputting whether the sentence is a summary-related sentence; then iteratively training the text summarization model with the classifier using the training set, and in each training step, on the one hand, optimizing the sentence encoding layer and the classification layer using a first optimizer and a summary sentence classification loss, and on the other hand, optimizing the encoder and decoder using a second optimizer and a summary text loss;

[0011] S4. Input the target text to be summarized into the trained text summary generation model. First, the encoder generates a text representation. Then, the text representation is input into the sentence encoding layer to generate a sentence representation. The text representation and sentence representation are fused and passed into the decoder. The decoder generates a summary corresponding to the target text.

[0012] Preferably, in S1, the original text of the sample is segmented into sentences using Stanford Parser and converted into JSON format for storage.

[0013] Preferably, in S1, for each sample, the original text and the summary text are compared for sentence similarity using the ROUGE tool, and 3 to 5 sentences that are most similar to the summary sentence are selected from the original text and marked as summary-related sentences. Each word in the summary-related sentence is marked as 1, and each word in the sentence in the original text that is not a summary-related sentence is marked as 0.

[0014] Preferably, in the training process of the text summary generation model, for the original text after sentence segmentation, [CLS] is inserted at the end of each sentence and then converted into an encoder input, the text representation is learned by the encoder, and then the learned text representation is input into the sentence encoding layer sentence by sentence to learn the sentence representation, and a label y∈{0,1} is assigned to each sentence in the original text by the classifier to indicate whether the sentence is a summary-related sentence, thereby converting the extractive summary generation task into a sentence classification task; the text representation output by the encoder and the sentence representation output by the sentence encoding layer are spliced ​​and passed into the decoder to perform the summary generation task.

[0015] Preferably, the generative pre-trained language model is Bart.

[0016] Preferably, the first optimizer selects the Adam optimizer with β1=0.9 and β2=0.999, and the learning rate is set to 2 -3 min(step -0. ,step·10000 -1. ); The second optimizer selects the Adam optimizer with β1=0.9 and β2=0.999, and the learning rate is set to 0.1 min (step -0. ,step·20000 -1. ), where step is the current training step of the model.

[0017] Preferably, the sentence encoding layer is implemented using a 2-layer Transformer.

[0018] Preferably, the classification layer adopts a sigmoid classifier.

[0019] Preferably, the summary-related sentence label loss adopts a binary cross entropy loss function.

[0020] Preferably, the summary label loss adopts a label smoothed cross entropy loss function.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] Compared with the existing technology, by introducing a sentence encoding layer and a classification layer between the encoder and decoder of the pre-trained language model, the generative and extractive text summarization are integrated, and the sentences as summaries are used to control and guide the generation process to improve the quality of text summarization. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flowchart of the text summary generation method that combines generative and extractive methods.

[0024] Figure 2 Schematic diagram of the abstract generation process of the method of the present invention.

[0025] Figure 3 This is a diagram of the text summary generation model structure after adding a classifier in the training phase of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1As shown, in a preferred embodiment of the present invention, a text summary generation method combining generative and extractive methods is provided. The method first extracts sentences containing important information in the text (called summary-related sentences) and then generates the required summary based on the summary-related sentences. Figure 2 The schematic diagram of the summary generation process is shown. It first screens the original text for content, finds summary-related sentences containing important information, and then generates the summary. It should be noted that the summary-related sentences here refer to some original sentences in the original text, but directly summarizing these original sentences will produce problems such as logical incoherence. Therefore, the summary in the present invention is not directly generated by summarizing these summary-related sentences, but requires the decoder to perform the generative summary generation task and regenerate the summary text. The steps of the text summary generation method combining generative and extractive methods are described in detail below, and the process is shown in S1 to S4:

[0028] S1: Obtain the dataset required for text summarization training (such as CNN / DailyMail, XSUM, etc.). Each sample in the dataset contains the original text and the summary text. Perform sentence segmentation on the original text and summary in each sample, and calculate the similarity between each sentence in the original text and each sentence in the summary text. Assign summary-related sentence labels to original text sentences with similarity above a threshold, so that each sample in the training set contains both the summary and the summary sentence label.

[0029] In the embodiment of the present invention, in S1 above, both the original text and the summary text of the sample are segmented using StanfordParser and converted to JSON format for storage. For each sample, the ROUGE tool is used to perform a pairwise sentence similarity comparison between the original text and the summary text. The 3-5 sentences most similar to the summary sentence are selected from the original text and labeled as summary-related sentences. Preferably, 3 sentences are selected from each original text as summary-related sentences. In this embodiment, selecting the {0, 1} label indicates that each sentence in the original text is labeled (words in summary-related sentences are labeled as 1, and words that are not summary-related sentences are labeled as 0).

[0030] S2: Based on the generative pre-trained language model, a sentence encoding layer is cascaded between the encoder and decoder of the generative pre-trained language model to construct a text summary generation model; in the text summary generation model, the sentences in the original text are input into the encoder, the text representation generated by the encoder is input into the sentence encoding layer to generate a sentence representation, the text representation generated by the encoder and the sentence representation generated by the sentence encoding layer are fused and input into the decoder, and the decoder generates a summary corresponding to the original text.

[0031] S3. Add a classification layer for auxiliary training on the sentence encoding layer in the text summary generation model, the sentence encoding layer and the classification layer constitute a classifier, classify each sentence in the original text based on the sentence representation generated by the sentence encoding layer, and output whether the sentence is a summary-related sentence; then use the training set to iteratively train the text summary generation model with the classifier, and in each training step, on the one hand, it is necessary to optimize the sentence encoding layer and the classification layer through the first optimizer and the summary sentence classification loss, and on the other hand, it is necessary to optimize the encoder and decoder through the second optimizer and the summary text loss.

[0032] The structure of the above text summary generation model is as follows Figure 3 As shown. The generative pre-trained language model is Bart, and the sentence encoding layer can be implemented using a two-layer Transformer. In the above text summary generation model training process, the original text after sentence segmentation in a single sample is represented as [sent1, sent2,…, sent m ], where sent i It represents the i-th sentence of the original text, and the original text contains a total of m sentences. After inserting [CLS] at the end of each sentence, it is converted into the encoder input. The encoder is used to learn the text representation, and then the learned text representation is input into the sentence encoding layer sentence by sentence to learn the sentence representation. The classification layer assigns a label y∈{0,1} to each sentence in the original text to indicate whether the sentence is a summary sentence, thereby converting the extractive summary generation task into a sentence classification task; the text representation output by the encoder and the sentence representation output by the sentence encoding layer are concatenated and fused and then passed into the decoder for the summary generation task.

[0033] In the above text summarization model, the encoder selects the original encoder and decoder of the language model. The sentence encoding layer cascaded between the end of the encoder and the front end of the decoder can capture document-level and sentence-level features to extract summaries. The process in the lth layer Transformer is expressed as:

[0034]

[0035]

[0036] Where h0=x is the input vector, LN is the layer normalization operation, and MHAtt is the multi-head attention operation used to focus on different aspects of the sentence. The superscript l represents the number of stacking layers. Thus, the top encoder generates a vector t for each token. i FFN represents the feed-forward network layer. l = 1,…,L, where L is the number of Transformer layers selected in the sentence encoding layer.

[0037] Add a sigmoid classifier to the final output of the sentence encoding layer: You can get represents the sentence representation of the final output of the i-th sentence, is the predicted value of the summary-related sentence label of the i-th sentence.

[0038] The sentence encoding layer in this invention can be optimized and adjusted based on actual needs. It can capture document-level features to extract important sentences. In this embodiment, the sentence encoding layer is a two-layer Transformer. Furthermore, the classification layer in this invention can use either a sigmoid classifier or a softmax classifier; in this embodiment, a sigmoid classifier is selected.

[0039] In the present invention, after the above model is initialized, two optimizers need to be set to adapt to the double optimization to achieve the purpose of adapting the pre-trained module and the non-pre-trained module, thereby integrating the extractive text summary into the generative text summary. The specific optimizer and loss function form can also be adjusted according to actual conditions. In this embodiment, the first optimizer selects the Adam optimizer with β1=0.9 and β2=0.999, and the learning rate is set to lr=2e -3 min(step -0.5 ,step·warmup -1.5 ), where warmup = 10,000. The second optimizer selects the Adam optimizer with β1 = 0.9 and β2 = 0.999, and the learning rate is set to lr = 0.1 min (step -0.5 ,step·warmup -1.5 ), where warmup = 20,000, where step is the current number of training steps of the model. This allows the encoder to extract important information faster, making the decoder optimization more stable.

[0040] The remaining hyperparameters during training can also be adjusted and optimized based on actual conditions. For example, in the subsequent examples, when using CNN / DailyMail and XSUM as training and test datasets, for the generative summarization task, for the CNN / DM dataset, the length penalty is set to 2.0, the maximum length is 142, the minimum length is 56, and the beam search is set to 4; for the XSUM dataset, the length penalty is set to 1.0, the maximum length is 62, the minimum length is 11, and the beam search is set to 6.

[0041] In this embodiment, the summary-related sentence label loss adopts a binary cross-entropy loss function (corresponding to the extractive text summary, calculated by the true value of the summary-related sentence label assigned in S1 and the summary-related sentence label prediction value generated by the above classifier), and the summary label loss adopts a label smoothing cross-entropy loss function (corresponding to the generative text summary, calculated by the true value of the summary label assigned in S1 and the summary label prediction value generated by the above decoder).

[0042] In order to avoid overfitting, the present invention can periodically save the corresponding model parameters during the training process, and then select the ROUGE index as the criterion for selecting the optimal model. The ROUGE-n index is:

[0043]

[0044] The ROUGE_L indicator is (L stands for the longest common subsequence):

[0045]

[0046]

[0047]

[0048] Where X represents the standard answer reference (i.e., summary), and Y represents the model prediction value. m represents the length of Y, and n represents the length of Y. In an embodiment of the present invention, Rouge2 can be selected as the model screening indicator. The best model selected is used for practical application or testing in step S4, and the Rouge1, Rouge2, and RougeL indicators are calculated using the ROUGE tool.

[0049] S4. Input the target text to be summarized into the trained text summary generation model. First, the encoder generates a text representation. Then, the text representation is input into the sentence encoding layer to generate a sentence representation. The text representation and sentence representation are fused and passed into the decoder. The decoder generates a summary corresponding to the target text.

[0050] The text summary generation method of the generative pre-trained language model described in S1 to S4 above is applied to a specific example to demonstrate its specific implementation method and technical effect.

[0051] Example

[0052] As shown in Table 1, this example uses CNN / DailyMail and XSUM as training and test datasets to verify the experimental results. These three datasets are single-text news summary datasets with different writing and summary styles (important information in the text is concentrated at the beginning or evenly distributed in the source document). The basic information of the three datasets is as follows:

[0053] Table 1: Basic information of the dataset

[0054]

[0055]

[0056] In this embodiment, the text summarization method of the generative pre-trained language model described in S1 to S4 above (denoted as Ours) is applied to the aforementioned dataset. At the same time, for four methods in the prior art (BERTSUMABS, PTGEN, PTGEN+COV, BERTSUMEXABS), the specific experimental results are shown in Tables 2 and 3:

[0057] Table 2: Experimental results on CNN / DM

[0058]

[0059] Table 3: Experimental results on XSUM

[0060]

[0061] As can be seen from Tables 2 and 3, extractive text summarization can generate summaries with higher Rouge-1 and Rouge-2 index values, outperforming other existing models. This demonstrates that the present invention addresses the problem of generative summarization by integrating extractive text summarization into the training of the summary generation task. By controlling and guiding the summary generation process through existing labeled data, the proposed method can more effectively utilize labeled data resources and improve model capabilities.

[0062] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A text summarization method combining generative and extractive methods, characterized in that: Here are the steps: S1: Obtain the dataset required for text summarization training. Each sample in the dataset contains the original text and the summary text. Segment the original text and summary in each sample into sentences, and calculate the similarity between each sentence in the original text and each sentence in the summary text. Assign summary-related sentence labels to original text sentences with similarity above a threshold, so that each sample in the training set contains both the summary and the summary sentence label. S2: Based on the generative pre-trained language model, a sentence encoding layer is cascaded between the encoder and decoder of the generative pre-trained language model to construct a text summarization model; in the text summarization model, sentences in the original text are input into the encoder, the encoder generates text representations, and then input into the sentence encoding layer to generate sentence representations. The text representations generated by the encoder and the sentence representations generated by the sentence encoding layer are fused and input into the decoder, which generates a summary corresponding to the original text; S3. Adding a classification layer for auxiliary training to the sentence encoding layer in the text summarization model, wherein the sentence encoding layer and the classification layer constitute a classifier, classifying each sentence in the original text based on the sentence representation generated by the sentence encoding layer, and outputting whether the sentence is a summary-related sentence; then iteratively training the text summarization model with the classifier using the training set, and in each training step, on the one hand, optimizing the sentence encoding layer and the classification layer using a first optimizer and a summary sentence classification loss, and on the other hand, optimizing the encoder and decoder using a second optimizer and a summary text loss; S4. Input the target text to be summarized into the trained text summary generation model. First, the encoder generates a text representation. Then, the text representation is input into the sentence encoding layer to generate a sentence representation. The text representation and sentence representation are fused and passed into the decoder. The decoder generates a summary corresponding to the target text.

2. The text summarization method combining generative and extractive methods as described in claim 1, characterized in that: In S1, the original text of the sample is segmented into sentences using Stanford Parser and converted into JSON format for storage.

3. The text summarization method combining generative and extractive methods as described in claim 1 is characterized in that: In S1, for each sample, the original text and the summary text are compared for sentence similarity using the ROUGE tool, and 3 to 5 sentences that are most similar to the summary sentence are selected from the original text and marked as summary-related sentences. Each word in the summary-related sentence is marked as 1, and each word in the sentence in the original text that is not a summary-related sentence is marked as 0.

4. The text summarization method combining generative and extractive methods as claimed in claim 1, characterized in that: During the training process of the text summary generation model, for the original text after sentence segmentation, [CLS] is inserted at the end of each sentence and then converted into an encoder input. The encoder is used to learn text representation, and the learned text representation is then input into the sentence encoding layer sentence by sentence to learn sentence representation. The classifier assigns a label y∈{0,1} to each sentence in the original text to indicate whether the sentence is a summary-related sentence, thereby converting the extractive summary generation task into a sentence classification task. The text representation output by the encoder and the sentence representation output by the sentence encoding layer are concatenated and passed to the decoder to perform the summary generation task.

5. The text summarization method combining generative and extractive methods as claimed in claim 1, characterized in that: The generative pre-trained language model is Bart.

6. The text summarization method combining generative and extractive methods as claimed in claim 1, characterized in that: The first optimizer selects the Adam optimizer with β1=0.9 and β2=0.999, and the learning rate is set to 2e -3 min(step -0.5 ,step·10000 -1.5 ); The second optimizer selects the Adam optimizer with β1=0.9 and β2=0.999, and the learning rate is set to 0.1 min (step -0.5 ,step·20000 -1.5 ), where step is the current training step of the model.

7. The text summarization method combining generative and extractive methods as claimed in claim 1, characterized in that: The sentence encoding layer is implemented using a 2-layer Transformer.

8. The text summarization method combining generative and extractive methods as claimed in claim 1, characterized in that: The classification layer adopts a sigmoid classifier.

9. The text summarization method combining generative and extractive methods as claimed in claim 1, characterized in that: The summary-related sentence label loss adopts a binary cross entropy loss function.

10. The text summarization generation method based on the combination of generative and extractive methods according to claim 1, characterized in that: The summary label loss adopts the label smoothed cross entropy loss function.

Images