Chinese long text-oriented hybrid automatic abstract generation method

Abstract

The invention discloses a Chinese long text-oriented hybrid automatic abstract generation method, and relates to the technical field of natural language processing. According to the method, an extraction type abstract method in an unsupervised mode is taken as a first stage, the complexity of a model is reduced, meanwhile, information loss is avoided, and long text content is compressed. Then, a generative abstract module is constructed based on comparative learning, document-level coding representation containing context semantic information is fused into the training process of the model, so that the model effectively guides the generated abstract to be compared with a standard abstract text, irrelevant information is prevented from being generated, and the problem of exposure deviation generally existing in the generative text abstract is solved; and the abstract generation quality is improved while the model complexity is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a hybrid automatic summary generation method for Chinese long texts. Background Art

[0002] In the digital age, the explosive growth of information has further aggravated the difficulty of quickly and accurately obtaining key information from massive data. As an important branch of natural language processing, automatic summarization technology aims to condense lengthy text content into short summaries that retain the core meaning of the original text. It is widely used in information retrieval, intelligent assistants, and news reports. According to different task processes, automatic text summarization methods are divided into three categories: extractive text summarization, generative text summarization, and hybrid text summarization. Although the extractive text summarization method has low complexity and can quickly obtain available key information, it lacks the ability to refine and generate the overall semantics of a text, and also has shortcomings such as semantic incoherence. The process and logic of the generative text summarization method to form a summary are closer to the way artificial summaries are generated. The language generation model can generate higher quality summaries when it fully understands the semantic information of the text. The generative text summarization method based on deep learning has developed into the mainstream of automatic text summarization. This type of method mainly adopts the encoder-decoder architecture and uses the attention mechanism to establish the relationship between the input sequence and the target sequence. However, due to the limited memory capacity of deep neural network models, it is difficult to learn long-range semantic information for long text input sequences, resulting in the model being unable to generate high-quality summaries. Although large language models based on Transformer can usually effectively capture the features of text and generate good summaries, the model training time of such architectures is usually positively correlated with the length of the sequence. In order to balance hardware resources and control training time, the length of the input sequence is limited to a certain extent. For long text sequences, the commonly used strategy is to truncate the input text. During the model input process, the model will truncate and ignore the text content that exceeds the maximum length limit of the model. This strategy has shown good results on many Chinese and English datasets. However, from the perspective of the currently available Chinese and English datasets, most of them are short text types, and news datasets account for a large proportion. The key information of news data is often concentrated at the beginning of the text, and the use of truncation does not affect its effect. However, texts in different fields have their own structural characteristics, and not all key information is concentrated at the beginning of the text. Therefore, when using generative text summarization methods to process long text data in other fields, truncation strategies often lead to serious information loss problems and exposure bias, which ultimately affects the quality of summary generation.

[0003] Based on the above, hybrid text summarization methods came into being, such as Figure 1The hybrid text summarization method shown in the figure combines the advantages of both extractive summarization and generative summarization, and often adopts a two-stage processing method. The first stage performs the extractive text summarization task, and then uses the output of the first stage as the input of the second stage to perform the generative text summarization task. The hybrid text summarization method combines the extractive summarization and generative summarization methods to achieve complementary advantages and overcome the limitations of a single method, thereby improving the accuracy and credibility of the summary content and obtaining a high-quality summary. Summary of the invention

[0004] In view of the shortcomings of the prior art in summarizing long Chinese texts, the present invention provides a hybrid automatic summary generation method for long Chinese texts to solve the problems of poor summary quality, high model complexity and high training cost generated by the existing automatic text summarization methods.

[0005] A hybrid automatic summary generation method for Chinese long text, the specific steps include:

[0006] Step 1: Construct a text summary dataset; the text summary dataset includes several Chinese long texts in the form of documents and corresponding standard summary texts;

[0007] Step 2: preprocessing the Chinese long text in the document form in the text summary dataset to obtain a preprocessed text summary dataset; the preprocessing includes sentence segmentation and character replacement of special characters in the Chinese long text in the document form;

[0008] Step 3: Divide the text summary dataset into training set and test set according to the set ratio;

[0009] Step 4: Build a hybrid text summarization model;

[0010] The hybrid text summarization model includes an extractive summarization module and a generative summarization module;

[0011] The extractive summarization module generates a preliminary summary result of each Chinese long text using an unsupervised extractive summarization method;

[0012] The generative summary module completes the generation of the final summary based on the preliminary summary result generated by the extractive summary module;

[0013] The generative summarization module includes three components: a pre-trained BERT encoder, a GPT-2 encoder, and a decoder;

[0014] The input of the pre-trained BERT encoder is the preliminary summary result generated by the extractive summarization module, which is input into the BERT layer to obtain a vector representation containing contextual semantic information;

[0015] The GPT-2 encoder performs deep semantic extraction and encoding on the text vector representation obtained by converting the input preliminary summary result generated by the extractive summarization module to obtain a character-level vector sequence of the fused text context; the text vector representation includes a word vector, a position vector and a segment vector, and the word vector, the position vector and the segment vector are randomly initialized according to the set dimensions and continuously optimized through the model training process;

[0016] The input of the decoder is the character-level vector sequence of the fused text context output by the GPT-2 encoder and the output of the decoder at the previous time step. After decoding based on the Transformer decoder architecture, a classifier component with a linear layer is introduced to complete the prediction of the output characters of the corresponding time step, thereby obtaining the final summary; the decoder is a Transformer decoder structure; the input of the decoder at each time step in the training phase is the standard summary text and the character-level vector sequence of the fused text context output by the GPT-2 encoder, and the input of the decoder at the previous time step is the character-level vector sequence of the fused text context output by the GPT-2 encoder; the test phase is the output of the decoder at the previous time step and the character-level vector sequence of the fused text context output by the GPT-2 encoder;

[0017] The decoder adopts a top-k and top-p joint sampling strategy for decoding;

[0018] Step 5: According to the Chinese long texts and standard summary texts in the form of documents in the training set, the loss function of the hybrid text summary model is constructed based on the idea of ​​contrastive learning, and the constructed hybrid text summary model is trained using the gradient descent algorithm to obtain the trained hybrid text summary model;

[0019] A pre-trained BERT encoder is used to obtain a vector representation containing contextual semantic information, and the vector representation containing contextual semantic information is integrated into the training process of the hybrid text summarization model. The document-level global semantic representation vector generated by the decoder is mapped to the space where the vector representation containing contextual semantic information generated by the BERT encoder is located. The hybrid text summarization model is measured by the semantic similarity between the standard summary text and the summary generated by the decoder, and the hybrid text summarization model is effectively guided to generate summary results that are increasingly similar to the standard summary text; the document-level global semantic representation vector is the output of the softmax layer of the decoder;

[0020] At each time step of the decoder, the similarity between the document-level global semantic representation vector of the hybrid text summary model and the vector representation containing contextual semantic information is calculated, and this similarity is added to the training process of the hybrid text summary model. After adding the similarity, the loss function of the hybrid text summary model is defined as

[0021] The loss function of the hybrid text summarization model is:

[0022]

[0023] Among them, λ is a hyperparameter, and Specifically:

[0024]

[0025] Where X represents the original Chinese long text input, Y = {y1, y2, ..., y |Y|} represents the standard summary text, |Y| represents the length of the standard summary text, t represents the time step, y t Characters representing the standard summary text input at time step t; y 0:t-1 represents the output sequence of characters predicted by the decoder from time step 0 to t-1, H t represents the hidden layer vector representation of the hidden layer output of the GPT-2 encoder at time step t; W represents the trainable parameter matrix of the GPT-2 encoder;

[0026]

[0027] Among them, α is a hyperparameter, sim(O t ,D) represents the similarity between the document-level global semantic representation vector output by the decoder softmax layer and the vector representation containing contextual semantic information. The specific calculation formula is as follows:

[0028]

[0029] Among them, O t represents the document-level global semantic representation vector output by the softmax layer of the decoder, D is the vector representation containing contextual semantic information, and O i is the document-level global semantic representation vector O t The i-th element in, n is the dimension, D i is the i-th element in D, a vector representing contextual semantic information;

[0030] Step 6: Use the Chinese long text in the form of documents in the test set to verify the trained hybrid text summary model, and adjust the parameters of the hybrid text summary model according to the verification results, and retrain the hybrid text summary model until the verification effect on the test set meets the requirements, and obtain the final hybrid text summary model;

[0031] Step 7: Input the long Chinese text to be summarized into the final hybrid text summary model to automatically generate the text summary result.

[0032] Compared with the prior art, the present invention has the following benefits:

[0033] The present invention proposes a hybrid automatic summary generation method for Chinese long texts. It uses an unsupervised extractive summary method as the first stage to reduce the complexity of the model while avoiding information loss and compressing the long text content. Then, a generative summary module is constructed based on contrastive learning to integrate the document-level encoding representation containing contextual semantic information into the model training process, so that the model can effectively guide the generated summary to be compared with the standard summary text, avoid generating irrelevant information, solve the exposure bias problem commonly existing in generative text summaries, and improve the quality of summary generation while reducing the complexity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of a hybrid text summarization method in the prior art;

[0035] Figure 2 It is a structural diagram of the hybrid text summarization model in an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of the structure of the generative summary module in an embodiment of the present invention;

[0037] Figure 4 This is a GPT framework diagram in an embodiment of the present invention;

[0038] Figure 5 Schematic diagram of input of a generative summary module in an embodiment of the present invention;

[0039] Figure 6 A schematic diagram of a decoding process in an embodiment of the present invention;

[0040] Figure 7 Schematic diagram of ROUGE-Recall value of ablation experiment in an embodiment of the present invention;

[0041] Figure 8 Schematic diagram of ROUGE-Precision value of ablation experiment in an embodiment of the present invention;

[0042] Fig. 9 Schematic diagram of ROUGE-F1 value of ablation experiment in an embodiment of the present invention;

[0043] Fig.10 Schematic diagram of ROUGE value of ablation experiment in an embodiment of the present invention;

[0044] Fig.11 This is a graph showing the change of the loss function in an embodiment of the present invention;

[0045] Fig.12 This is a comparison diagram of loss function changes in an embodiment of the present invention;

[0046] Fig.13 This is an example of the summary generation result of the NLPCC2017 single document text summary dataset in an embodiment of the present invention;

[0047] Fig.14 This is an example of the summary generation result of the CAIL2020 judicial summary dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0049] A hybrid automatic summary generation method for Chinese long text, the specific steps include:

[0050] Step 1: Construct a text summary dataset; the text summary dataset includes several Chinese long texts in the form of documents and corresponding standard summary texts;

[0051] The research on automatic text summarization for Chinese started relatively late, and the development of summary datasets in the field of Chinese long texts is also relatively slow. In the direction of automatic summary generation in Chinese, the datasets with high recognition include Tsinghua News (THUCNews), NLPCC2017 single document text summary dataset, LCSTS dataset, Sohu News dataset, and CAIL2020 judicial summary dataset. The THUCNews dataset was compiled by the Natural Language Processing Laboratory of Tsinghua University based on the historical data of Sina News, and contains more than 800,000 news documents. The NLPCC2017 single document text summary dataset is the dataset used for Task 3 of the Chinese Computing Conference Competition organized by the CCF Chinese Information Technology Committee in 2017, containing 50,000 standard and non-standard summary samples. The LCSTS dataset is a large-scale Chinese short text summary dataset compiled by Harbin Institute of Technology based on Sina Weibo, containing more than two million real short texts. The Sohu News dataset contains about 1.4 million news texts and title data from multiple channels of Sohu News. CAIL (Challenge of AI in Law) is a Chinese legal intelligent technology evaluation competition held by China Legal Intelligent Technology Evaluation to promote the application of artificial intelligence technologies such as natural language processing in the legal field. The CAIL2020 Judicial Summary Dataset is a text summary dataset for the 2020 Judicial Summary track. Among the above Chinese datasets, most of them are short text summary datasets, except for the CAIL2020 Judicial Summary Dataset and the NLPCC2017 Single Document Text Summary Dataset. Therefore, the present invention selects the CAIL2020 Judicial Summary Dataset and the NLPCC2017 Single Document Text Summary Dataset to evaluate the model.

[0052] The first dataset is the CAIL2020 Judicial Abstracts dataset. During the experiment, we expanded the CAIL2020 Judicial Abstracts dataset to obtain a larger dataset, which includes 13,531 annotated civil judgments. This dataset is a truly long text dataset. The average character length of the dataset body is 2468, of which the maximum character length is 20,832 and the minimum character length is 882; the average character length of the standard abstract is 280, of which the maximum character length is 1594 and the minimum character length is 66. Its statistical information is shown in Table 1.

[0053] The second dataset is the NLPCC2017 single document text summary dataset, which contains a total of 50,000 Chinese single document news summaries. In the NLPCC2017 single document text summary dataset, the average character length of the text is 990, of which the maximum character length is 22,312 and the minimum character length is 52; the average character length of the standard summary is 45, of which the maximum character length is 128 and the minimum character length is 17. Compared with the CAIL2020 judicial summary dataset, although the average character length of the text is close to 1,000, it is still not a particularly long sequence. The dataset text is preprocessed according to the model requirements. The statistical information of the preprocessed CAIL2020 judicial summary dataset is shown in Table 1.

[0054] Table 1 Dataset character length statistics

[0055]

[0056] Step 2: Preprocess the Chinese long text in the document form in the text summary dataset to obtain a preprocessed text summary dataset; the preprocessing includes sentence segmentation and character replacement of special characters in the Chinese long text in the document form to avoid noise signals affecting the reliability of subsequent encoding results;

[0057] The special characters in this embodiment include: punctuation marks, numbers, dates, HTML tags and other non-text characters, which are replaced accordingly according to the specific characters predefined in the vocabulary.

[0058] Step 3: Divide the text summary dataset into training set and test set according to the set ratio;

[0059] In this implementation, the preprocessed CAIL2020 judicial summary dataset and the NLPCC2017 single document text summary dataset are divided. Since the scale of the dataset is not large, the training set and the test set are divided in a ratio of 7:3, where 10% of the training set is taken as the validation set. The specific division of the dataset is shown in Tables 2 and 3.

[0060] Table 2 Division of CAIL2020 judicial summary dataset

[0061]

[0062] Table 3 NLPCC2017 single document text summary dataset division

[0063]

[0064] Step 4: Combining the advantages of extractive text summarization and generative summarization, construct Figure 2 The hybrid text summarization model shown;

[0065] The hybrid text summarization model includes an extractive summarization module and a generative summarization module;

[0066] The extractive summarization module generates a preliminary summary result of each Chinese long text using an unsupervised extractive summarization method;

[0067] The generative summary module completes the generation of the final summary based on the preliminary summary result generated by the extractive summary module;

[0068] like Figure 3 As shown, the generative summary module includes three components: a pre-trained BERT encoder, a GPT-2 encoder, and a decoder;

[0069] The input of the pre-trained BERT encoder is the preliminary summary result generated by the extractive summarization module, which is input into the BERT layer to obtain a vector representation containing contextual semantic information;

[0070] The GPT-2 encoder is Figure 4 As shown in , the text vector representation obtained by converting the preliminary summary result generated by the extractive summary module is deeply semantically extracted and encoded to obtain a character-level vector sequence of the fused text context; as shown in Figure 5 As shown, the text vector representation includes a word vector, a position vector and a segment vector, and the word vector, the position vector and the segment vector are randomly initialized according to the set dimensions and then continuously optimized through the model training process;

[0071] like Figure 6As shown, the input of the decoder is the character-level vector sequence of the fused text context output by the GPT-2 encoder and the output of the decoder at the previous time step. After decoding based on the Transformer decoder architecture, a classifier component with a linear layer is introduced to complete the prediction of the output characters of the corresponding time step, thereby obtaining the final summary; the decoder is a Transformer decoder structure; the input of the decoder at each time step in the training phase is the standard summary text and the character-level vector sequence of the fused text context output by the GPT-2 encoder, and the input of the decoder at the previous time step is the character-level vector sequence of the fused text context output by the GPT-2 encoder; the test phase is the output of the decoder at the previous time step and the character-level vector sequence of the fused text context output by the GPT-2 encoder;

[0072] The decoder adopts a top-k and top-p joint sampling strategy for decoding; top-k sampling randomly samples from the k words with the highest probability at each time step, without considering other low-probability words. This method can avoid the model outputting content that is irrelevant to the original text during decoding, and can also make the output summary more flexible. In addition, this method dynamically adjusts the number of candidate words according to different input texts, and can also control the generation process by adjusting the size of k. It can also be used in combination with other decoding strategies to further optimize the results of summary generation. Top-p sampling is also called core sampling. The algorithm idea is to randomly sample only from the minimum set of words whose cumulative probability exceeds a certain threshold at each time step, focusing only on the core part of the probability distribution, and ignoring other low-probability words, avoiding irrelevant words generated by the model, while maintaining the flexibility of the output sequence.

[0073] Step 5: According to the Chinese long texts and standard summary texts in the form of documents in the training set, the loss function of the hybrid text summary model is constructed based on the idea of ​​contrastive learning, and the constructed hybrid text summary model is trained using the gradient descent algorithm to obtain the trained hybrid text summary model;

[0074] The present invention integrates the contrastive learning mechanism into the generative summarization module. In the field of automatic text summarization, there is a natural association of semantic similarity between the input text and the generated summary. This advantage means that the problem can be easily modeled as a contrastive learning mechanism. In the training phase, the semantic similarity between the summary generated by the decoder and the standard summary text is maximized to alleviate the exposure bias problem of the generative summarization module.

[0075] Specifically, a pre-trained BERT encoder is used to obtain a vector representation containing contextual semantic information, the vector representation containing contextual semantic information is integrated into the training process of the hybrid text summarization model, the document-level global semantic representation vector generated by the decoder is mapped to the space where the vector representation containing contextual semantic information generated by the BERT encoder is located, and a semantic similarity measurement model between the standard summary text and the summary generated by the decoder is used to effectively guide the model to generate summary results that are increasingly similar to the standard summary text; the document-level global semantic representation vector is the output of the softmax layer of the decoder;

[0076] At each time step of the decoder, the similarity between the document-level global semantic representation vector of the hybrid text summary model and the vector representation containing contextual semantic information is calculated, and this similarity is added to the training process of the hybrid text summary model to alleviate the exposure bias problem when the model generates summaries. After adding the similarity, the loss function of the hybrid text summary model is defined as L. The semantic similarity of the text is intuitively the degree of semantic similarity between sentences. The evaluation of semantic similarity usually relies on advanced algorithms and technologies. The present invention uses cosine similarity to measure the similarity between the document-level global semantic representation vector and the vector representation containing contextual semantic information;

[0077] The loss function of the hybrid text summarization model is:

[0078] L=L a +λL s

[0079] Among them, λ is a hyperparameter, and Specifically:

[0080]

[0081] Where X represents the original Chinese long text input, Y = {y1, y2, ..., y |Y|} represents the standard summary text, |Y| represents the length of the standard summary text, t represents the time step, y t Characters representing the standard summary text input at time step t; y 0:t-1 represents the output sequence of characters predicted by the decoder from time step 0 to t-1, H t represents the hidden layer vector representation of the hidden layer output of the GPT-2 encoder at time step t; W represents the trainable parameter matrix of the GPT-2 encoder;

[0082]

[0083] Among them, α is a hyperparameter, sim(O t,D) represents the similarity between the document-level global semantic representation vector output by the decoder softmax layer and the vector representation containing contextual semantic information. The specific calculation formula is as follows:

[0084]

[0085] Among them, O t represents the document-level global semantic representation vector output by the softmax layer of the decoder, D is the vector representation containing contextual semantic information, and O i is the document-level global semantic representation vector O t The i-th element in, n is the dimension, D i is the i-th element in D, a vector representing contextual semantic information;

[0086] Step 6: Use the Chinese long text in the form of documents in the test set to verify the trained hybrid text summary model, and adjust the parameters of the hybrid text summary model according to the verification results, and retrain the hybrid text summary model until the verification effect on the test set meets the requirements, and obtain the final hybrid text summary model;

[0087] Step 7: Input the long Chinese text to be summarized into the final hybrid text summary model to automatically generate the text summary result.

[0088] The model of the present invention uses the PyTorch framework for model construction, and uses the pre-trained model of the GPT2-Chinese version as the basic architecture. GPT2-Chinese obtains 100G of high-quality Chinese corpus by data cleaning of the Chinese part of CommonCrawl, and trains the final model on this corpus. It is a general pre-trained model specifically for the Chinese field. After fine-tuning, the present invention uses it as the basic architecture of the generative summary module. The specific model parameter settings are shown in Table 4, and the experimental environment of the present invention is shown in Table 5.

[0089] Table 4 Model parameter setting table

[0090]

[0091]

[0092] In order to verify the effectiveness of the model of the present invention in the automatic summarization task, a series of experiments were carried out on the CAIL2020 judicial summary dataset. When the important parameters were set consistently, the experimental results of the model of the present invention were compared with those of the following mainstream models on the dataset. The comparison results are shown in Table 6.

[0093] (1) TextRank: It is a classic graph-based text summarization algorithm. Based on the PageRank algorithm, it models the relationship between words or sentences in the text and calculates the importance of sentences through weighted graph iteration. It is a widely used and efficient algorithm in the field of automatic text summarization.

[0094] (2) BertSum: This model was proposed by Liu et al. in 2019. It fine-tuned the pre-trained model BERT to obtain deep text features and combined it with the text summary classification task for summary extraction, achieving good results.

[0095] (3) LSTM+Attention: This model is an improvement on the classic sequence-to-sequence model. It uses BiLSTM as the encoder to capture the long-distance dependencies and position information of the text, and then uses unidirectional LSTM as the decoder, and combines it with the attention mechanism to build a summary generation model.

[0096] (4) PGN: This model is a generative text summarization model proposed by Abigail et al. by combining the traditional Seq2Seq model with Pointer-Generator Networks (PNG). The model improves the sequence-to-sequence model in two orthogonal ways. The addition of a pointer-generator network expands the model's vocabulary and determines whether to generate words in the vocabulary or directly copy words in the original text based on the probability distribution of words in the input text.

[0097] (5)PGN+law: This model is a method for generating judicial judgment summary based on a pointer generation network and integrating legal provisions proposed by Wei et al. in 2023. This model innovatively introduces legal provisions as external knowledge to assist judgment based on the characteristics of legal texts in the judicial field, deeply encodes the relevant legal provisions involved in the judgment documents, and combines the pointer generation network to retain key information. This model is completely targeted at the CAIL2020 judicial summary dataset and is a vertical field-oriented method. In addition, the length setting of the input text and the maximum length setting of the generated summary during the experiment are basically consistent with this model, and the comparability is strong.

[0098] Table 6 Experimental results on CAIL2020 judicial summary dataset

[0099]

[0100]

[0101] As can be seen from Table 6, the model of the present invention has been improved to varying degrees in most indicators compared with other mainstream models in terms of evaluation indicators ROUGE-1, ROUGE-2 and ROUGE-L. Compared with the pointer generation network model PNG+law that integrates legal provisions, the model of the present invention has improved the ROUGE-1 / 2 scores by 12.37 / 7.87% in the F1 indicator, respectively, indicating that the model of the present invention can more accurately capture important words and phrases in text summaries, and is better than the PNG+law model in matching words and word combinations; but the ROUGE-L score in the F1 indicator has dropped by 2.86%, indicating that the model of the present invention does not consider the contextual consistency and coherence when generating summaries, resulting in poor long-distance word matching. The reason may be that the semantic incoherence caused by the text compression of Chinese long text data in the extractive summary stage, which in turn affects the understanding deviation of the text semantics in the generative summary stage. In addition, since the PNG+law model is a model that introduces external knowledge, its model parameters are set with 100 characters of external legal knowledge, so it performs better in terms of long-distance dependencies. Compared with the text summary model BertSum based on the pre-trained model BERT, the ROUGE-L score of the model of the present invention is improved by 2.43% in the F1 indicator. Compared with the model based on Seq2Seq and attention mechanism, the ROUGE-L score of the model of the present invention is improved by 3.58% in the F1 indicator. Compared with the pointer generation network model PNG, the ROUGE-L score of the model of the present invention is improved by 1.05% in the F1 indicator, indicating that although the model of the present invention has shortcomings in dealing with the problem of long-distance dependencies in generating summaries, it is still improved to a certain extent compared with the mainstream models. In summary, the model of the present invention performs well in capturing local information.

[0102] In order to further evaluate the summary generation quality of the model proposed in this invention, a number of experiments were conducted on the NLPCC2017 single document text summary dataset, and the results of various experimental evaluation indicators were compared with the evaluation results of the following mainstream models on this dataset. The comparison results are shown in Table 7. The two-stage contrastive learning model Cons-2 proposed by Yang et al. was selected as the baseline model. To ensure the fairness of the comparative experimental results, the model of this invention sets the maximum length of the input sequence to 1024 and limits the maximum length of the generated summary to 60. The purpose is to be consistent with the experimental parameters of Yang et al., provide a relatively consistent experimental environment, and ensure that the evaluation of model performance is comparable and scientific.

[0103] (1)RNN: This model is based on a recurrent neural network, but does not introduce an attention mechanism. The last hidden layer of the encoder is used as the input of the decoder to generate a summary.

[0104] (2)RNN-context: This model introduces a text summary generation model based on semantic relevance, recurrent neural network and attention mechanism.

[0105] (3) GPT: This model is based on the GPT-2 architecture. During the training phase, no pre-trained model is used for fine-tuning. Instead, the model is randomly initialized and then retrained. The same training parameters are set as the model of the present invention.

[0106] (4) CopyNet: This model is based on the Seq2Seq network and attention mechanism. It adds a generation and copy mechanism on the decoder side. The fragments in the input sequence are selectively copied to the output sequence, which has achieved good results in experiments.

[0107] (5) DRGD: This model proposes a deep recurrent model based on the Seq2Seq network and attention mechanism. It uses a recursive latent random model to learn potential structural information and integrates it into the generative summary model, thereby improving the quality of sentences generated by the generative model.

[0108] (6) R-NET: This model is a deep learning model that uses a self-matching network for understanding.

[0109] (7) KIGN: This model uses keyword encoding as a guiding signal in the summary generation model. By expanding the pointer generation network, it realizes the bidirectional replication of input sentences and keywords and achieves good model performance.

[0110] (8) PEGASUS: This model is pre-trained based on the Transformer encoder-decoder. During the pre-training stage, the sentences in the input text are blocked or deleted, and other sentences are used to predict the generation results.

[0111] (9) Cons-2: This model is based on a two-stage contrastive learning model framework to advance summary generation tasks. Through continuous comparison and optimization, the model has achieved good experimental results on the NLPCC2017 single document text summary dataset.

[0112] Since the Cons-2 model uses the F1 value of the ROUGE index as the evaluation index of the experimental results, for the convenience of comparison, the experiment in this section adopts the same method. The experimental results of the model of the present invention on the NLPCC2017 single document text summary dataset are shown in Table 7. It can be seen that compared with other mainstream models under the evaluation index, the F1 value index has been improved to varying degrees. Compared with the Chinese automatic text summarization method Con-2 based on two-stage contrastive learning, the model of the present invention has improved the Rouge-1 / 2 / L scores by 6.03 / 6.42 / 6.38% on the F1 index, respectively, indicating the effectiveness of the model of the present invention. In addition, compared with other mainstream models, the model of the present invention has advantages in ROUGE evaluation indicators to varying degrees. Due to the large amount of data in the NLPCC2017 single document text summary dataset, the model has a more obvious improvement in the ROUGE index on this dataset, indicating that sufficient training data is conducive to the model to obtain better optimization effects. In addition, it can be seen that when the GPT architecture model does not adopt the pre-trained model fine-tuning and comparative learning strategy, facing the same model settings, the final ROUGE index has a large gap with the model of the present invention, which shows the effectiveness of the pre-trained BERT encoder in the model of the present invention.

[0113] Table 7 Experimental results of NLPCC2017 single document text summary dataset

[0114]

[0115] In order to fully verify the effectiveness of each module in the model proposed in the present invention, an ablation experiment was conducted. On the CAIL2020 judicial summary dataset, a total of three ways of generating summaries were tested. For the hybrid automatic summary generation method for Chinese long text proposed in the present invention, the text compression ratio ratio of the extractive summary stage is set to 0.4, so as to ensure that the input text length of the generative summary stage does not exceed the maximum input length limit of the model to avoid information loss. The three parts of the ablation experiment all use the same model settings. In order to ensure the consistency of the ablation experiment parameters, during the ablation experiment, the same settings as the previous experiment are adopted to avoid affecting the results of the final ablation experiment due to changes in model parameters. Usually, the size of the ROUGE value is positively correlated with the length of the sequence. Longer sequences will lead to more n-gram overlaps or longer common subsequences, thereby making the ROUGE value higher. Therefore, the ablation experiment does not pursue the SOTA performance of the summary generation results, but only verifies the effectiveness of the module. The maximum input sequence length of the model is set to 512, the maximum output length of the model summary is set to 150, and the epochs during model training are all 40. The ablation experiment results are as follows Figures 7 to 9 shown.

[0116] Figures 7 to 9The model effects after removing different modules on the CAIL2020 judicial summary dataset are shown respectively. In order to make the comparison clearer, this section uses visualization and charts to intuitively show the changes in ROUGE values ​​under different circumstances. The Our model in the figure is the model proposed by the present invention, Our-pretraining is a GPT-2 model that only uses a 6-layer Transformer and is trained from scratch, and Our-CL represents the model of the present invention without contrast learning. When the above three models are trained and evaluated with the same parameters on the CAIL20200 judicial summary dataset, it is found that the overall performance of the model of the present invention in the ROUGE indicator is the highest, and regardless of the Recall value, Precision value or F1 value, the indicator change trends of the three models are basically the same, which fully proves the effectiveness of each module of the model of the present invention. The pre-training model of the GPT2-Chinese version used in the model of the present invention is a general Chinese pre-training model specifically for the Chinese field, and the representation of Chinese semantics is clearer. It is not difficult to find that there is a small gap between the Our-pretraining model and the Our-CL model in some indicators, indicating that the contrastive learning mechanism proposed in the present invention improves the model effect to a similar degree as the pre-training model trained with large-scale Chinese corpus improves the model of the present invention, which indirectly illustrates the effect of the contrastive learning mechanism.

[0117] Since the CAIL20200 judicial summary dataset is small in scale and has a small amount of data, in order to fully verify the effect of the model of the present invention, an ablation experiment was also conducted on the NLPCC2017 single document text summary dataset. In order to verify the impact of the pre-training model on the summary generation results and the training process, the model of the present invention trained two models separately with the same parameters. Among them, the maximum input length of the model is set to 1024, the maximum output length of the target summary is set to 60, and the training epochs are set to 20. Our-pretraining model is a pre-training model that does not use the GPT2-Chinese version, but only uses a model with a 6-layer Transformer GPT architecture for training; Our model is a generative summary model proposed by the present invention that uses a pre-training model and comparative learning. In addition, since the maximum input length and the maximum output length of the summary of the baseline model Cons-2 during the experiment are the same as those of the model of the present invention, it is also added to the comparative experiment. The specific experimental result indicators are as follows Fig.10 shown.

[0118] from Fig.11 and Fig.12As can be seen, the loss function of the pre-trained model converges faster and fits the data better, proving that the fine-tuning mode of the pre-trained model can learn deeper semantic representations and better understand the meaning of the input text and its contextual relationship, thereby generating higher quality summaries.

[0119] The actual summary generation results of the model of the present invention in the NLPCC2017 single document text summary dataset are as follows: Fig.13 As shown in , it can be seen that the output results of the model of the present invention are relatively consistent with the content of the original text. In addition, the summary output by the model of the present invention also contains more key information, describes the specific content of the action, and is relatively close to the content of the standard summary. The results output by Our-Pretraining, which does not use a pre-trained model, do not describe the key information of the original text, the information is not accurate enough, and lacks specific information related to the original text. In addition, ChatGPT 3.5 was used to generate a summary of the original text. It can be seen that the summary generated by ChatGPT is also relatively concise and clear, but the information provided in terms of the rectification object is less, and it focuses more on the time requirements and disposal measures of the action.

[0120] The actual summary generation results of the proposed model in the CAIL2020 judicial summary dataset are shown in Figure 2. Fig.14 As shown in the figure, the original text is a case of "Bank Loan Contract Dispute", and the character length of the original text reaches 2294 characters. The general generative summary model is difficult to handle such long sequence texts, and usually adopts a stage discarding strategy. The summary generation results of the model of the present invention are shown in the table. Compared with the standard summary, the summary generation results of the model of the present invention basically cover most of the content of the standard summary. As shown in the bold keywords in the table, from the summary of the case to the application of each law, the output of the model of the present invention is consistent with the standard summary. In addition, ChatGPT 3.5 was also used to generate a summary of the original text on the CAIL2020 judicial summary dataset. It can be seen that although the summary content generated by ChatGPT is correct in the summary and characterization of the event, the summary content has no focus, but simply lists several details in the original text.

Claims

1. A hybrid automatic summary generation method for Chinese long text, characterized in that: The specific steps include: Step 1: Construct a text summary dataset; the text summary dataset includes several Chinese long texts in the form of documents and corresponding standard summary texts; Step 2: Preprocess the Chinese long text in the document form in the text summary dataset to obtain the preprocessed text summary dataset; Step 3: Divide the text summary dataset into training set and test set according to the set ratio; Step 4: Build a hybrid text summarization model; Step 5: According to the Chinese long texts and standard summary texts in the form of documents in the training set, the loss function of the hybrid text summary model is constructed based on the idea of ​​contrastive learning, and the constructed hybrid text summary model is trained using the gradient descent algorithm to obtain the trained hybrid text summary model; Step 6: Use the Chinese long text in the form of documents in the test set to verify the trained hybrid text summary model, and adjust the parameters of the hybrid text summary model according to the verification results, and retrain the hybrid text summary model until the verification effect on the test set meets the requirements, and obtain the final hybrid text summary model; Step 7: Input the long Chinese text to be summarized into the final hybrid text summary model to automatically generate the text summary result.

2. A hybrid automatic summary generation method for Chinese long text according to claim 1, characterized in that: The preprocessing in step 2 includes sentence segmentation and character replacement of special characters in the Chinese long text in the document form.

3. A hybrid automatic summary generation method for Chinese long text according to claim 1, characterized in that: The hybrid text summarization model in step 4 includes an extractive summarization module and a generative summarization module; The extractive summarization module generates a preliminary summary result of each Chinese long text using an unsupervised extractive summarization method; The generative summary module completes the generation of the final summary based on the preliminary summary result generated by the extractive summary module; The generative summarization module includes three components: a pre-trained BERT encoder, a GPT-2 encoder, and a decoder; The input of the pre-trained BERT encoder is the preliminary summary result generated by the extractive summarization module, which is input into the BERT layer to obtain a vector representation containing contextual semantic information; The GPT-2 encoder performs deep semantic extraction and encoding on the text vector representation obtained by converting the input preliminary summary result generated by the extractive summarization module to obtain a character-level vector sequence of the fused text context; the text vector representation includes a word vector, a position vector and a segment vector, and the word vector, the position vector and the segment vector are randomly initialized according to the set dimensions and continuously optimized through the model training process; The input of the decoder is the character-level vector sequence of the fused text context output by the GPT-2 encoder and the output of the decoder at the previous time step. After decoding based on the Transformer decoder architecture, a classifier component with a linear layer is introduced to complete the prediction of the output characters of the corresponding time step, thereby obtaining the final summary; the decoder is a Transformer decoder structure; the input of the decoder at each time step in the training phase is the standard summary text and the character-level vector sequence of the fused text context output by the GPT-2 encoder, and the input of the decoder at the previous time step is the character-level vector sequence of the fused text context output by the GPT-2 encoder, and the test phase is the output of the decoder at the previous time step and the character-level vector sequence of the fused text context output by the GPT-2 encoder.

4. A hybrid automatic summary generation method for Chinese long text according to claim 3, characterized in that: The decoder adopts a top-k and top-p joint sampling strategy for decoding.

5. The hybrid automatic summary generation method for Chinese long text according to claim 1 is characterized in that: The step 5 is specifically as follows: A pre-trained BERT encoder is used to obtain a vector representation containing contextual semantic information, and the vector representation containing contextual semantic information is integrated into the training process of the hybrid text summarization model. The document-level global semantic representation vector generated by the decoder is mapped to the space where the vector representation containing contextual semantic information generated by the BERT encoder is located. The hybrid text summarization model is measured by the semantic similarity between the standard summary text and the summary generated by the decoder, and the hybrid text summarization model is effectively guided to generate summary results that are increasingly similar to the standard summary text; the document-level global semantic representation vector is the output of the softmax layer of the decoder; At each time step of the decoder, the similarity between the document-level global semantic representation vector of the hybrid text summary model and the vector representation containing contextual semantic information is calculated, and this similarity is added to the training process of the hybrid text summary model. After adding the similarity, the loss function of the hybrid text summary model is defined as The loss function of the hybrid text summarization model is: Among them, λ is a hyperparameter, and Specifically: p(y t |y 0:t-1 ,X)=Softmax(WH t ) Where X represents the original Chinese long text input, Y = {y1, y2, ..., y |Y| } represents the standard summary text, |Y| represents the length of the standard summary text, t represents the time step, y t Characters representing the standard summary text input at time step t; y 0:t-1 represents the output sequence of characters predicted by the decoder from time step 0 to t-1, H t represents the hidden layer vector representation of the hidden layer output of the GPT-2 encoder at time step t; W represents the trainable parameter matrix of the GPT-2 encoder; Among them, α is a hyperparameter, sim(O t ,D) represents the similarity between the document-level global semantic representation vector output by the decoder softmax layer and the vector representation containing contextual semantic information. The specific calculation formula is as follows: Among them, O t represents the document-level global semantic representation vector output by the softmax layer of the decoder, D is the vector representation containing contextual semantic information, and O i is the document-level global semantic representation vector O t The i-th element in, n is the dimension, D i is the i-th element in D, a vector containing contextual semantic information.

Images