A text summarization generation method and system based on contrast learning
By optimizing the generative text summarization model using a contrastive learning approach, and combining MLE loss and contrastive loss functions, the exposure bias problem in generative text summarization is solved, thereby improving the quality and consistency of the generated text summaries.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2023-06-19
- Publication Date
- 2026-06-02
Smart Images
Figure CN116737919B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing technology, specifically relating to a text summarization method and system based on contrastive learning. Background Technology
[0002] Currently, deep learning-based generative text summarization is typically treated as a sequence-to-sequence (Seq2Seq) problem, generating summaries in an autoregressive manner. Such models are usually trained using maximum likelihood estimation (MLE) based on a teacher-forcing approach. This introduces a gap between the loss function and the evaluation metric, because the loss function is based on local token-level predictions, while the evaluation metric needs to compare the overall similarity between the reference summary and the model output. Furthermore, during the testing phase, the model needs to autoregressively generate output, meaning that errors from the previous step accumulate. This gap between training and testing has been termed exposure bias in previous work.
[0003] Convolutional neural networks pre-trained using contrastive learning methods have advanced computer vision object detection and image segmentation. Recently, contrastive learning has been introduced into several conditional text generation tasks, such as machine translation and text summarization. Contrastive learning focuses on learning common features among similar instances and distinguishing differences between dissimilar instances. Compared to generative learning, contrastive learning does not need to focus on the intricate details of instances; it only needs to learn to distinguish data at the abstract semantic level of feature space. Therefore, the model and its optimization become simpler, and its generalization ability is stronger. Summary of the Invention
[0004] The purpose of this invention is to provide a text summarization method and system based on contrastive learning to solve the technical problem of exposure bias in generative text summarization in the prior art.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] In a first aspect, the present invention provides a text summarization method based on contrastive learning, comprising:
[0007] S1: Obtain the dataset needed to generate the text summary, and preprocess the samples in the dataset;
[0008] S2: Load the preprocessed samples into the preset model for fine-tuning;
[0009] S3: Iteratively train the fine-tuned model to obtain the final model;
[0010] S4: Input the text into the final model to generate a text summary.
[0011] Optionally, the dataset includes several samples, each containing text and a reference summary. Each sample is preprocessed to obtain a preprocessed training set, test set, and validation set.
[0012] Optionally, the preprocessing of samples in the dataset specifically includes:
[0013] Sub-word segmentation of the samples was performed to build a corresponding vocabulary.
[0014] Insert a start marker at the beginning of the reference abstract and an end marker at the end. Truncate samples longer than 512 characters and pad samples shorter than 512 characters.
[0015] The text and its reference abstract are encoded to obtain an encoded representation.
[0016] Optionally, the preset model is based on the Transformer architecture and is constructed from an encoder and a decoder. Loading the training set and validation set into the preset model for training and validation includes:
[0017] The preprocessed training set is loaded into a pre-defined model, and the model is fine-tuned based on maximum likelihood estimation using the validation set. Specifically:
[0018] The original documents in the training set are input into the encoder to obtain their context representations. The obtained context representations and reference summaries are input into the decoder to obtain the hidden state of the decoder.
[0019] The decoder is then passed through a fully connected layer and a softmax function to obtain the probability distribution of the words. Finally, the MLE loss function is obtained by cross-entropy with the reference summary.
[0020] The model is iteratively trained using the training and validation sets based on the MLE loss function to obtain a fine-tuned model.
[0021] Optionally, the iterative training of the fine-tuned model to obtain the final model further includes:
[0022] Load the fine-tuned model, read the training set, generate multiple candidates for each sample in the training set using the model, and sort each candidate according to the sequence-level evaluation index relative to the reference abstract to obtain the sorted candidate abstract;
[0023] The hidden state of the encoder of each sample is obtained by passing it through the encoder of the model. The sorted candidate summary and the hidden state of each sample are input into the decoder of the model to calculate the score of the candidate summary. The contrastive loss function of the sample is obtained by combining the sorting of the candidate summaries.
[0024] The model is iteratively trained by combining the training set and validation set with the contrastive loss function and the MLE loss function.
[0025] Optionally, the iterative training of the model by combining the contrastive loss function and the MLE loss function includes:
[0026] The final model is obtained by iteratively training the final loss function on the test set. The final loss function is the sum of the contrastive loss and the MLE loss.
[0027] Optionally, the sorted candidate summaries are obtained in the following way:
[0028] During the model generation process, multiple candidate summaries are obtained through diversified bundle search, and the average value of Rouge-1 and Rouge-2 is selected as the evaluation index to rank the candidate summaries.
[0029] Optionally, the score of the candidate summary is obtained in the following way:
[0030] Input the candidate digest into the decoder;
[0031] In the decoder, the probability distribution is obtained through a fully connected layer and the softmax function is used to calculate the score of the candidate summary by performing a cross-entropy function with the candidate summary.
[0032] Optionally, the text is input into the trained final model, and the optimal text summary is generated and selected by the model's autoregression using a diversified bundle search.
[0033] Secondly, the present invention also provides a text summarization generation system based on contrastive learning, characterized in that it includes: a dataset sample preprocessing module, a text summarization generation module, and a text summarization output module; wherein:
[0034] The dataset sample preprocessing module is used to obtain the dataset required for text summarization and to preprocess the samples in the dataset.
[0035] Text summarization generation module: Used to input text into the final model and generate a text summary;
[0036] Text summarization output module: Used to output the generated text summaries.
[0037] The beneficial effects and advantages of this invention are as follows:
[0038] This contrastive learning-based text summarization method fine-tunes a pre-trained model and then uses it as a re-ranker to sort candidate summaries based on evaluation metrics, yielding a contrastive loss. A multi-task learning approach is employed, combining Mean Leakage (MLE) and contrastive loss to optimize the model. MLE loss ensures word-level prediction accuracy, while contrastive loss assigns higher generation probabilities to candidate summaries with higher evaluation scores. Furthermore, online sampling is used during the contrastive learning phase to generate candidate summaries, ensuring consistency between the model as a generator and the re-ranker, and continuously improving the quality of candidate summaries to ensure the learnability of the ranking task objective. Attached Figure Description
[0039] Figure 1 This is a schematic diagram of the process of the present invention;
[0040] Figure 2 This is a schematic diagram of the two-stage training of the model of the present invention;
[0041] Figure 3 This is a schematic diagram of the experimental results of each model on the ROUGE index in the embodiments of the present invention. Detailed Implementation
[0042] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0043] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.
[0044] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0045] One embodiment, such as Figures 1 to 3 As shown, a text summarization method based on contrastive learning is provided, which includes the following steps to alleviate the exposure bias problem in generative text summarization:
[0046] S1: Obtain the dataset needed to generate the text summary, and preprocess the samples in the dataset;
[0047] S2: Load the preprocessed samples into the preset model for fine-tuning;
[0048] S3: Iteratively train the fine-tuned model to obtain the final model;
[0049] S4: Input the text into the final model to generate a text summary.
[0050] refer to Figure 1 and Figure 2 As shown, in this embodiment, an English text summarization dataset (Xsum) is first obtained, which includes several samples, each of which contains text and a reference summary. Each sample is preprocessed to obtain a preprocessed training set, test set, and validation set.
[0051] The preprocessing includes: sub-word segmentation of the samples, establishment of a corresponding vocabulary, insertion of a start marker at the beginning of the reference abstract and an end marker at the end, truncation of samples longer than 512 characters and padding of samples shorter than 512 characters, resulting in a preprocessed file.
[0052] Then download and load the pre-trained model. In this embodiment, the pre-trained model is preferably based on the Transformer architecture and is constructed by an encoder and a decoder. Load the pre-processed training set and validation set into the pre-trained model and fine-tune the pre-trained model based on maximum likelihood estimation.
[0053] Let X be the source document of the training set in the preprocessed file, Y be the reference summary, y be the reference summary length, and T be the encoder. Encoder The decoder is T Decoder Input the source document into encoder T Encoder The hidden state h = T is obtained.Encoder , hide state h enc and reference summary are input to decoder T Decoder At time t, where (0 <= t < len) y The probability distribution of the predicted words:
[0054]
[0055] In the formula: W and b are the parameters of the fully connected layer, y ＜t These are the words preceding time t in the reference abstract;
[0056] The probability distribution is The final MLE loss is L MLE =CrossEntrypy(P vocb (Y), then based on L MLE The model is fine-tuned by performing multiple iterations of training.
[0057] The fine-tuned trained model is loaded, and n candidate summaries S are obtained for each source document X in the preprocessed file through diversified beam search. cans ={s1, s2, ... s n Let R be the value of the i-th candidate digest Rouge-1 and Rouge-2. 1i ,R 2i The overall score is R. i =(R 1i +R 2i ) / 2, according to R i The candidate abstracts are sorted by size to obtain the sorted candidate abstracts. Where k i S represents sorted_cans The index of the i-th candidate summary in the original candidate summary set;
[0058] The source document X is passed through the model encoder to obtain the encoder hidden state h. enc =T Encoder (x), for each sorted candidate summary and h enc via T Decoder Obtain the hidden state h of the decoder dec =T Decoder (Y, h) enc The final score is calculated as follows:
[0059]
[0060] in Indicates length penalty, The length of , α is a hyperparameter, w and b are the parameters of the fully connected layer, and the candidate summary score set is . The contrastive learning loss function is:
[0061]
[0062] Where γ is a hyperparameter;
[0063] Combining L from the above MLE The final loss function is:
[0064] L tot =L MLE +βL ctr
[0065] For the final loss function L tot Perform multiple iterations of training to obtain the final model and save it;
[0066] The text is input into the final saved model, and the optimal summary is generated by the model's autoregression using a diversified bundle search.
[0067] This embodiment also discloses a text summarization generation system based on contrastive learning, including: a dataset sample preprocessing module, a text summarization generation module, and a text summarization output module; wherein:
[0068] The dataset sample preprocessing module is used to obtain the dataset required for text summarization and to preprocess the samples in the dataset.
[0069] Text summarization generation module: Used to input text into the final model and generate a text summary;
[0070] Text summarization output module: Used to output the generated text summaries.
[0071] The specific workflows and structures of the dataset sample preprocessing module, text summarization generation module, and text summarization output module have been described in detail above, so they will not be repeated here.
[0072] refer to Figure 2 As shown, the word-level prediction accuracy of the model is improved by fine-tuning the model based on the MLE loss function in the first stage. In the second stage, the model first acts as a generative model to sample and generate candidate summaries online. Then, the model acts as a reorderer to sort the candidate summaries. The evaluation index scores of the candidate summaries and the contrastive loss function are combined. Finally, the model is iteratively trained by combining the MLE loss function and the contrastive loss function.
[0073] To illustrate the applicability and superiority of the present invention, the publicly available English dataset Xsum was selected as the experimental data. This dataset consists of BBC articles and single-sentence summaries, and the training set, validation set, and test set contain 204,045 (90%), 11,332 (5%), and 11,334 (5%) documents, respectively.
[0074] The model of this invention uses T5-small based on the Transformer structure as the encoder and decoder. The input length of the encoder is limited to 512, and the output length of the decoder does not exceed 128.
[0075] In the first stage of model training, the AdaFactor optimizer is used with a learning rate set to 2×1e. -5 The size, number of iterations, and batch size are set to 60 and 32, respectively. In the second phase of training, the same optimizer is used with a learning rate of 1e. -5 The sampling size is 8, the number of iterations is set to 15, the batch size is 6, and γ is set to 0.075.
[0076] During the inference phase, the model length penalty is set to 0.6, and a diversified bundle search is adopted as the decoding strategy, with the bundle search size set to 8.
[0077] The experiment uses the ROUGE automatic summarization evaluation algorithm as the evaluation index. This index measures the quality of computer-generated text summarization by calculating the overlap between the automatic summary to be evaluated and the standard human summary on the n-gram.
[0078] The calculation method for ROUGE-N is as follows:
[0079]
[0080] Where: Count match(n-gram) This represents the frequency of an n-gram that exists in both the automatically generated text summary and the standard human summary. In contrast, Count(n-gram) represents the frequency of an n-gram that exists only in the standard reference summary.
[0081] The comparative model used in the experiment is as follows:
[0082] T5-MLE: A pre-trained model with the same model structure as the model, fine-tuned based on MLE loss;
[0083] T5-ConSum: A contrastive learning scheme that generates candidate summaries through offline sampling, using the candidate and reference summaries as negative and positive samples, respectively, to optimize the model. The original paper was based on BART; this invention, to maintain consistency in the validation comparison, uses the T5-small model.
[0084] Ours: The solution proposed in this invention;
[0085] The experimental results of each model on the ROUGE evaluation index are as follows: Figure 3 As shown, the model proposed in this invention has achieved optimal results on ROUGE-1, ROUGE-2, and ROUGE-L.
[0086] In summary, this invention proposes a generative text summarization method based on contrastive learning to improve the exposure bias problem in generative text summarization. The model is first fine-tuned as a generative model based on maximum likelihood estimation (MLE) loss. Then, the model is used as a reorderer to rank the candidate summaries generated by the model according to the evaluation index to obtain the contrastive loss. The model is optimized by combining the MLE loss function and the contrastive loss function through multi-task learning.
[0087] The main idea of this method is to ensure word-level prediction accuracy through the MLE loss function, while using the contrastive loss function to make the model assign candidate summaries with higher evaluation index scores and higher generation probability. In addition, during the contrastive learning stage, this invention uses online sampling to generate candidate summaries to ensure the consistency between the model as a generator and the re-ranker, while continuously improving the quality of candidate summaries to ensure the learnability of the ranking task objective. Experimental results verify the feasibility and superiority of the proposed scheme.
[0088] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.
Claims
1. A text summarization method based on contrastive learning, characterized in that: include: S1: Obtain the dataset needed to generate the text summary, and preprocess the samples in the dataset; S2: Load the preprocessed samples into the preset model for fine-tuning; S3: Iteratively train the fine-tuned model to obtain the final model; S4: Input the text into the final model to generate a text summary; The preset model is based on the Transformer architecture and is constructed from an encoder and a decoder. The training set and validation set are loaded into the preset model for training and validation, including: The preprocessed training set is loaded into a pre-defined model, and the model is fine-tuned based on maximum likelihood estimation using the validation set. Specifically: The original documents in the training set are input into the encoder to obtain their context representations. The obtained context representations and reference summaries are input into the decoder to obtain the hidden state of the decoder. The decoder is then passed through a fully connected layer and a softmax function to obtain the probability distribution of the words. Finally, the MLE loss function is obtained by cross-entropy with the reference summary. The model is iteratively trained using the training and validation sets based on the MLE loss function to obtain a fine-tuned model. The iterative training of the fine-tuned model to obtain the final model also includes: Load the fine-tuned model, read the training set, generate multiple candidates for each sample in the training set using the model, and sort each candidate according to the sequence-level evaluation index relative to the reference abstract to obtain the sorted candidate abstract; The hidden state of the encoder of each sample is obtained by passing it through the encoder of the model. The sorted candidate summary and the hidden state of each sample are input into the decoder of the model to calculate the score of the candidate summary. The contrastive loss function of the sample is obtained by combining the sorting of the candidate summaries. The model is iteratively trained by combining the training set and validation set with the contrastive loss function and the MLE loss function. The iterative training of the model by combining the contrastive loss function and the MLE loss function includes: The final model is obtained by iteratively training the final loss function on the test set. The final loss function is the sum of the contrastive loss function and the MLE loss function. The sorted candidate summaries are obtained in the following way: During the model generation process, multiple candidate summaries are obtained through diversified bundle search, and the average value of Rouge-1 and Rouge-2 is selected as the evaluation index to rank the candidate summaries; The score of the candidate summary is obtained in the following way: Input the candidate digest into the decoder; In the decoder, the probability distribution is obtained through a fully connected layer and the softmax function is used to calculate the score of the candidate summary by performing a cross-entropy function with the candidate summary.
2. The text summarization method based on contrastive learning according to claim 1, characterized in that: The dataset includes several samples, each containing text and a reference summary. Each sample is preprocessed to obtain a preprocessed training set, test set, and validation set.
3. The text summarization method based on contrastive learning according to claim 2, characterized in that: The preprocessing of samples in the dataset specifically includes: Sub-word segmentation of the samples was performed to build a corresponding vocabulary. Insert a start marker at the beginning of the reference abstract and an end marker at the end. Truncate samples longer than 512 characters and pad samples shorter than 512 characters. The text and its reference abstract are encoded to obtain an encoded representation.
4. A text summarization method based on contrastive learning according to claim 1 or 3, characterized in that: The text is input into the trained final model, and the optimal text summary is generated and selected by the model's autoregression using a diversified bundle search.
5. A text summarization generation system based on contrastive learning, characterized in that, The text summarization method based on contrastive learning according to any one of claims 1 to 4 includes: a dataset sample preprocessing module, a text summarization generation module, and a text summarization output module; wherein: The dataset sample preprocessing module is used to obtain the dataset required for text summarization and to preprocess the samples in the dataset. Text summarization generation module: Used to input text into the final model and generate a text summary; Text summarization output module: Used to output the generated text summaries.