Ranking-based summary generation training method, system, and storage medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a summary generation training method and system based on ranking and a storage medium, the summary generation training method based on ranking comprises the following steps: obtaining training samples, the training samples comprising positive samples and negative samples, and combining the positive samples and the negative samples to construct training data; obtaining a scoring model, inputting the training data into the scoring model for training, and enabling the scoring model to score the factual consistency of the positive samples and the negative samples; obtaining a summary model and training the summary model, sampling the output of the summary model, obtaining multiple initial summaries with different generation probabilities for the same input, scoring the initial summaries based on the scoring model; ranking the initial summaries based on the generation probabilities and the scored initial summaries, calculating a loss function according to the ranking result, and training the summary model, so that the initial summaries with higher factual consistency scores have greater generation probabilities; and the method improves the factual consistency of the generated summaries.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a sorting-based summary generation training method, system, and storage medium. Background Technology

[0002] With the development of technology, the channels for communicating with customers through telephone, online text and third-party software are constantly increasing, resulting in a rapid increase in the amount of online data. After communicating with customers, users need to summarize the communication process to accurately summarize the content and form a summary for the subsequent use of the data.

[0003] Existing generative document summarization models typically adopt an encoder-decoder architecture. Since the model does not impose any constraints on the generated summary content, there is a certain probability that the generative summarization model will produce illusory content, that is, the generated content has not appeared in the original text, or contradicts real-world knowledge.

[0004] To address this issue, existing methods can be broadly categorized into three types. The first type utilizes output augmentation to help the model supplement factual consistency information in the document. This type of method typically requires additional input, which can be important sentences, entities, and relationships between entities in the original document. However, this method requires additional tools to parse the content of the input document, making the process of generating a summary cumbersome and impacting efficiency.

[0005] The second approach utilizes maximum likelihood estimation (MLE) as the cost function, which is commonly used for generative summarization models. However, this approach does not take into account the issue of factual consistency of the generated content. It requires reinforcement learning to introduce additional feedback information about the factual consistency of the summary in order to improve the model's performance. This approach is more difficult to train, the model is not conducive to convergence, and the choice of feedback information also seriously affects the model's performance.

[0006] The third approach involves post-processing to train a scoring function that determines the degree of entailment between the summary and the input document. This function selects the most relevant summary from among the various candidate summaries output by the generative summarization model using a beam search algorithm. While this method improves the overall model output, it does not alter the performance of the summarization model itself. Summary of the Invention

[0007] To address the issue that existing summarization models cannot simultaneously guarantee factual consistency and model performance, this invention provides a sorting-based summarization generation training method, system, and storage medium.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a sorting-based summary generation training method, comprising the following steps:

[0009] The process involves: acquiring training samples, including positive and negative samples, and constructing training data based on combinations of these samples; acquiring a scoring model by inputting the training data into the scoring model for training, enabling the scoring model to score the factual consistency of positive and negative samples; acquiring and training a summarizing model by sampling the output of the summarizing model to obtain multiple initial summaries with different generation probabilities from the same input, and scoring the initial summaries based on the scoring model; ranking the initial summaries based on their generation probabilities and scores, calculating a loss function based on the ranking results, and training the summarizing model so that the initial summaries with higher factual consistency scores have a greater generation probability; and finally, ranking the initial summaries based on their generation probabilities and scores, calculating a loss function based on the ranking results, and training the summarizing model so that the initial summaries with higher factual consistency scores have a higher generation probability.

[0010] The ranking is based on the generation probability and the initial summary after scoring. The specific formula for calculating the loss function based on the ranking result is as follows:

[0011]

[0012]

[0013] in, Indicates the first i Scoring of the initial summary; Indicates the first j Scoring of the initial summary; Indicates the first i The probability of generating an initial summary; Indicates the first j The probability of generating an initial summary; Indicates satisfaction , Indicates dissatisfaction .

[0014] Preferably, the positive samples include manually labeled sample data that meets factual consistency and is collected from existing databases.

[0015] Preferably, the method for obtaining negative samples specifically includes the following steps:

[0016] Obtain the source text, which includes the source summary and the source text. Tag the source text based on the word list annotation model, and tag the entity nouns or verbs in the source text and source summary respectively. Sample the entity nouns or verbs in the positive samples and select at least one entity noun or verb. Replace the selected entity nouns or verbs based on the tagged entity nouns or verbs to generate sample data with different content but the same part of speech.

[0017] Preferably, the scoring model is the BERT model.

[0018] Preferably, the scoring model is obtained, and the training data is input into the scoring model for training, so that the scoring model scores the factual consistency of positive and negative samples. Specifically, this involves:

[0019] The training data consists of sample pairs constructed from documents and summaries. The sample pairs are concatenated and input into the BERT model. The sample pairs are labeled 0-1 based on the BERT model, with positive samples labeled as 1 and negative samples labeled as 0.

[0020] Preferably, the process involves acquiring and training a summarizing model, sampling the output of the summarizing model to obtain multiple initial summaries with different generation probabilities for the same input, and scoring the initial summaries based on a scoring model. Specifically, this includes the following steps:

[0021] Obtain a summary model based on a general domain and train it using a cost function; sample the output of the summary model and obtain multiple initial summaries with different generation probabilities for the same input using a beam search algorithm; score the initial summaries based on a scoring model.

[0022] Preferably, the cost function for training the summarizing model is:

[0023]

[0024] in, Indicates the first i The probability of generating an initial summary.

[0025] To address the aforementioned technical problems, this invention provides another technical solution as follows: a sorting-based summary generation training system, comprising the following modules:

[0026] The system comprises the following modules: a construction module for acquiring training samples, including positive and negative samples, and constructing training data based on combinations of these samples; a processing module for acquiring a scoring model, inputting the training data into the scoring model for training, and ensuring that the scoring model scores positive samples higher than negative samples; and a generation module for acquiring and training a summarization model, sampling the output of the summarization model to obtain multiple initial summaries with different generation probabilities from the same input, and scoring the initial summaries based on the scoring model; sorting the initial summaries based on their generation probabilities and scores, calculating a loss function based on the sorting results, and training the summarization model so that the initial summaries with higher factual consistency scores have a higher generation probability.

[0027] The ranking is based on the generation probability and the initial summary after scoring. The specific formula for calculating the loss function based on the ranking result is as follows:

[0028]

[0029]

[0030] in, Indicates the first i Scoring of the initial summary; Indicates the first j Scoring of the initial summary; Indicates the first i The probability of generating an initial summary; Indicates the first j The probability of generating an initial summary; Indicates satisfaction , Indicates dissatisfaction .

[0031] To solve the above-mentioned technical problems, the present invention provides another technical solution as follows: a computer-readable storage medium storing a computer program, which, when executed, implements the sorting-based summary generation training method as described in any of the preceding claims.

[0032] Compared with existing technologies, the sorting-based summary generation training method, system, and storage medium provided by this invention have the following beneficial effects:

[0033] 1. A sorting-based summary generation method provided in this embodiment of the invention includes the following steps: First, training samples are obtained, including positive and negative samples, and training data is constructed based on the combination of positive and negative samples; then, a scoring model is obtained, and the training data is input into the scoring model for training, so that the scoring model scores the factual consistency of positive and negative samples; next, a summary model is obtained and trained, and the output of the summary model is sampled to obtain multiple initial summaries with different generation probabilities for the same input, and the initial summaries are scored based on the scoring model; finally, the initial summaries are sorted based on the generation probability and the scored initial summaries, a loss function is calculated based on the sorting result, and the summary model is trained so that the initial summaries with higher factual consistency scores have a higher generation probability; that is, a scoring model is trained by constructing training samples to score the factual consistency of the generated results, and then the output of the summary model is sampled. This method obtains multiple initial summaries with different generation probabilities from the same input, and sorts them according to their generation probabilities and scores to train the summarization model, resulting in an initial summary with a higher factual consistency score. This improves the factual consistency of the generated summaries and ensures the performance of the summarization model. Furthermore, this method has low training costs, does not require additional labeled data to train the scoring model, and its implementation uses both positive and negative samples to train the model more comprehensively, ensuring the quality and accuracy of the generated summaries. Scoring the factual consistency of positive and negative samples helps improve the model's ability to select training samples, resulting in greater factual consistency and more comprehensive, accurate, and targeted generated summaries. Additionally, sorting the initial summaries based on their generation probabilities and scores and then training the summarization model allows for more efficient acquisition of generated summaries that meet factual consistency requirements, and the training stability is good.

[0034] 2. The positive samples provided in this embodiment of the invention include manually labeled sample data that meets the requirements of factual consistency and is collected from existing databases. This setting ensures that the data of the positive samples is based on real facts and can provide a reliable data source. If there are no positive samples from factual sample data during the model training process, the factual consistency of the generated summary will be reduced, resulting in the generation of false information.

[0035] 3. The method for obtaining negative samples provided in this embodiment of the invention specifically includes the following steps: First, the source text is obtained, which includes the source summary and the source text. The source text is labeled based on the word list annotation model, and the entity nouns or verbs in the source text and the source summary are labeled respectively. Then, based on the entity nouns or verbs in the positive samples, at least one entity noun or verb is selected. Finally, based on the labeled entity nouns or verbs, the selected entity nouns or verbs are replaced to generate sample data with different content but the same part of speech. This setting can increase the diversity of data, improve the generalization ability of negative samples by replacing entity nouns and verbs, reduce data bias, and enable the subsequent scoring model to better understand and classify different types of summaries, thereby improving factual consistency. It can also reduce the repetition of samples and avoid overfitting of the model. For generating sample data with different content but the same part of speech, it can help the algorithm learn a wider range of semantic representations, which is beneficial to the generation of summaries and improves the generation quality.

[0036] 4. The scoring model provided in this embodiment of the invention trains the model by inputting training data into the scoring model, enabling the scoring model to score the factual consistency of positive and negative samples. Specifically, the training data consists of sample pairs constructed from documents and summaries. The sample pairs are concatenated and input into the BERT model. Based on the BERT model, the sample pairs are labeled with 0-1, with positive samples labeled as 1 and negative samples labeled as 0. This setting is used to judge whether the training samples meet the factual consistency requirement. It makes full use of the information of the sample pairs. By concatenating the sample pairs constructed from documents and summaries and inputting them into the model, the information of the two can be fused to more comprehensively represent the semantic features of the sample pairs, avoiding the model using single feature information. This helps the model improve the accuracy of distinguishing between positive and negative samples and ensures the performance of the scoring model.

[0037] 5. The present invention provides a method for obtaining and training a summarizing model, sampling the output of the summarizing model to obtain multiple initial summaries with different generation probabilities for the same input, and scoring the initial summaries based on a scoring model. Specifically, this includes the following steps: obtaining a summarizing model based on a general domain and training it using a cost function; sampling the output of the summarizing model and obtaining multiple initial summaries with different generation probabilities for the same input using a beam search algorithm; scoring the initial summaries based on a scoring model; the trained summarizing model can obtain high-quality initial summaries, while the beam search algorithm can generate multiple initial summaries, increasing the diversity of summaries, covering more summary content and styles, and providing more comprehensive information; further calculating the generation probability of the initial summaries allows for the evaluation of the rationality and accuracy of each initial summary.

[0038] 6. The present invention also provides a sorting-based summary generation training system and a computer-readable storage medium, which have the same beneficial effects as the sorting-based summary generation training method described above, and will not be repeated here. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of the steps of a sorting-based summary generation training method provided in the first embodiment of the present invention.

[0041] Figure 2 This is a flowchart of the negative sample acquisition step in step S1 of a sorting-based summary generation training method provided in the first embodiment of the present invention.

[0042] Figure 3 This is a flowchart of step S3 of a sorting-based summary generation training method provided in the first embodiment of the present invention.

[0043] Figure 4 This is an example of a sorting-based summary generation training method provided in the first embodiment of the present invention. Figure 1 .

[0044] Figure 5 This is an example of a sorting-based summary generation training method provided in the first embodiment of the present invention. Figure 2 .

[0045] Figure 6 This is a schematic block diagram of a sorting-based summary generation training system provided in the second embodiment of the present invention.

[0046] Explanation of reference numerals in the attached diagram:

[0047] 1. A sorting-based summary generation training system;

[0048] 10. Construction module; 11. Processing module; 12. Generation module. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] Please see Figure 1 The first embodiment of the present invention provides a sorting-based summary generation training method, comprising the following steps:

[0051] S1: Obtain training samples, which include positive and negative samples, and construct training data based on the combination of positive and negative samples;

[0052] It can be explained that positive samples refer to data that meets the requirement of factual consistency; conversely, negative samples refer to data that does not meet the requirement of factual consistency. Using a combination of positive and negative samples for training results in relatively balanced data, avoiding bias in model training caused by class imbalance, making the model evaluation more comprehensive, and ensuring the model's performance. It also ensures that the data order is random, preventing the model from learning patterns in the data and affecting its generalization ability.

[0053] S2: Obtain the scoring model, input the training data into the scoring model for training, and make the scoring model score the factual consistency of positive and negative samples.

[0054] It can be explained that the scoring model extracts features from the training data to help the model better learn the differences between positive and negative samples, enabling the scoring model to score the factual consistency of the training data; in addition, by training the scoring model, the model can make predictions and classifications on its own when dealing with new sample data, thus improving the performance of the scoring model.

[0055] S3: Obtain and train the summarization model, sample the output of the summarization model to obtain multiple initial summaries with different generation probabilities for the same input, and score the initial summaries based on the scoring model;

[0056] To explain, the process involves first acquiring and training a summarizing model, where the model is trained based on a general domain to establish a baseline. Then, the output of the summarizing model is sampled to obtain multiple initial summaries with different generation probabilities from the same input. This involves sampling training data that satisfies factual consistency into the baseline-established summarizing model, enabling it to generate multiple initial summaries with different generation probabilities. Next, the initial summaries are input into a scoring model to obtain assigned scores, which summarize the quality or relevance of the initial summaries for subsequent operations.

[0057] S4: Sort the initial summaries based on the generation probability and the score, calculate the loss function based on the ranking result, and train the summary model so that the initial summaries with higher fact consistency scores have a higher generation probability.

[0058] It can be explained that the initial summaries are sorted in descending order by generation probability and score, that is, the initial summaries with higher generation probability and score are ranked higher in the list; then the loss function is calculated based on the ranking result, and the summarization model is trained so that the initial summaries with higher fact consistency scores have a greater generation probability.

[0059] It should be noted that the generation probability of the summary model on the initial summary can represent the factual consistency information of the model; that is, the higher the generation probability of the initial summary, the higher its factual consistency.

[0060] Understandably, by training a scoring model using constructed training samples to score the factual consistency of the generated results, and then sampling the output of the summarizing model to obtain multiple initial summaries with different generation probabilities for the same input, and sorting them according to the generation probability and the scored initial summaries to train the summarizing model to obtain initial summaries with higher factual consistency scores, the factual consistency of the generated summaries can be improved, ensuring the performance of the summarizing model. In addition, this method has low training cost, does not require additional labeled data to train the scoring model, and its implementation process can train the model more comprehensively using positive and negative samples, ensuring the quality and accuracy of the generated summaries. Scoring the factual consistency of positive and negative samples helps to improve the model's ability to select training samples, making it have greater factual consistency, and the generated summaries can be more comprehensive, accurate and targeted. Furthermore, sorting the generated probabilities and the scored initial summaries and training the summarizing model can more efficiently obtain generated summaries that meet the factual consistency requirements, and the training stability is good.

[0061] Please combine Figure 1 and Figure 2 Furthermore, positive samples include manually labeled sample data that meets factual consistency, collected from existing databases. This setting ensures that the data of positive samples is based on real facts and can provide a reliable data source. If there are no positive samples derived from facts during the model training process, the factual consistency of the generated summary will be reduced, resulting in the generation of false information.

[0062] To better illustrate, human annotation refers to summaries written by humans based on their understanding of the document. These summaries are crucial for generating and evaluating summaries to improve model performance.

[0063] Understandably, positive samples are training data that ensures factual consistency. The reason why manually annotated summaries collected from existing databases are designated as positive samples is that summaries are based on human understanding of documents. That is, humans carefully read and understand the content of documents and extract the key points and important information from the summaries. This process takes place in a human subjective environment, and therefore will try to ensure that the information in the summaries is consistent with the facts in the original documents. In addition, although human summaries are subject to subjective influence, they are also reviewed and proofread to ensure their accuracy and consistency.

[0064] Furthermore, the method for obtaining negative samples in step S1 specifically includes the following steps:

[0065] S11: Obtain the source text, which includes the source summary and the source text. Tag the source text based on the word tagging model, and tag the entity nouns or verbs in the source text and the source summary respectively.

[0066] To better illustrate, a vocabulary tagging model refers to a natural language processing technique, typically used for text summarization and information extraction tasks. The main goal of this model is to tag key information or important content in a text based on a predefined vocabulary or keyword list.

[0067] Preferably, the vocabulary labeling model is the spaCy model, which is a general natural language processing library that can process large amounts of data in real time or in batches, thus improving the efficiency of model training.

[0068] Specifically, source text that satisfies factual consistency is obtained. The source text includes sample pairs of source summary and source text. Then, a vocabulary tagging model is used to mark the entity nouns or verbs in the source summary and source text for subsequent operations.

[0069] S12: Sample based on entity nouns or verbs in the positive samples, and select at least one entity noun or verb;

[0070] It can be explained that using entity nouns or verbs for labeling can provide important semantic information for model training, help the model better understand the structure and content of the input text, and make subsequent summary generation more accurate, complete, and logically coherent.

[0071] To better illustrate this, the present invention provides an embodiment where, if the marked entity noun is "bank," it can be identified as an important organization. Further association with its definition, establishment date, and other key information helps the model better learn the theme it conveys. If the marked verb is "training," it can be identified as a key step or process. By marking verbs, the model can learn important tasks or behaviors in the source text to grasp key information. In other words, marking entity nouns or verbs enables subsequent models to better understand the input, select more relevant and important information, and enhance the model's factual consistency. Furthermore, such marking improves the fluency and logic of subsequent operations, resulting in better stability during training.

[0072] S13: Based on the tagged entity nouns or verbs, replace the selected entity nouns or verbs to generate sample data with different content but the same part of speech.

[0073] It can be explained that such operations enhance the data, expand the training dataset, and enable subsequent training of the summarizing model to be more comprehensive.

[0074] Understandably, this setting increases data diversity, improves the generalization ability of negative samples by replacing entity nouns and verbs, reduces data bias, and enables subsequent scoring models to better understand and classify different types of summaries, thereby improving factual consistency. It also reduces sample repetition and avoids model overfitting. Furthermore, generating sample data with different content but the same part of speech helps the algorithm learn a wider range of semantic representations, which is beneficial for summary generation and improves generation quality.

[0075] To better illustrate this, the present invention provides an embodiment. Assume the source text is: "On March 12th, a certain continent's bank maintained its three major policy interest rates unchanged at its policy meeting and decided to increase asset purchases by an additional 120 billion euros until the end of the year. This day also became a global stock market 'circuit breaker day,' with stock markets in eight countries triggering circuit breakers simultaneously, including a certain stock experiencing its third circuit breaker in history. Why didn't the certain continent's bank cut interest rates? The president of the certain continent's bank stated at a press conference that fiscal policy should be the first and most important tool to respond to the crisis." The entity noun "certain continent" is marked in this example. Then, a positive sample is provided: "On March 12th, a certain country's stock market fell again. The president of a certain continent's bank said at a press conference that fiscal policy should be the first and most important tool to deal with the crisis." "A certain country" is selected as the entity noun. Based on the marked "certain continent" and "a certain country" in both examples, sample data with different content but the same part of speech is generated: "On March 12th, a certain continent's stock market fell again. The president of a certain continent's bank said at a press conference that fiscal policy should be the first and most important tool to deal with the crisis."

[0076] Furthermore, the scoring model is the BERT model.

[0077] To clarify, the BERT model is a pre-trained language model that can effectively learn the contextual information and semantic representation of a language through large-scale unsupervised training. Various pre-trained versions of the BERT model are available and can effectively understand and process texts in multiple languages, saving a significant amount of computing resources and time.

[0078] Furthermore, step S2 specifically includes:

[0079] The training data consists of sample pairs constructed from documents and summaries. The sample pairs are concatenated and input into the BERT model. The sample pairs are labeled 0-1 based on the BERT model, with positive samples labeled as 1 and negative samples labeled as 0.

[0080] It can be explained that by concatenating the text and the corresponding summary and inputting them into the BERT model, the semantic understanding capabilities of the BERT model can be used to capture the correlation and semantic similarity between the document and the summary, which is conducive to better differentiation in the future. In addition, concatenating the information of the two and outputting it to the BERT model can fuse the information, which can more comprehensively represent the semantic features of the sample pair. This avoids the model using only the single information of the document or the summary, but can consider the content of both at the same time, which is conducive to improving the accuracy of classification.

[0081] Next, the identified sample pairs are labeled with 0-1 using the BERT model, where positive samples are labeled with 1 and negative samples with 0. In other words, a classifier is built on the output of the BERT model, mapping the BERT output to binary probability values. The sample pairs are then input into the model for prediction, and the probability values ​​are between 0 and 1, which indicate the likelihood that the information belongs to a positive sample, and vice versa. This labeling of the data makes it easier for the scoring model to score positive and negative samples separately, and improves the accuracy of the scoring.

[0082] Understandably, this setting is used to judge whether the training samples meet the requirement of factual consistency. It makes full use of the information of the sample pairs, and inputs the sample pairs constructed from the document and the summary into the model. The fusion of the information of the two can more comprehensively represent the semantic features of the sample pairs, avoid the model using single feature information, help the model improve the accuracy of distinguishing between positive and negative samples, and ensure the performance of the scoring model.

[0083] Please combine Figure 1 and Figure 3 Furthermore, step S3 specifically includes the following steps:

[0084] S31: Obtain the summarization model, based on a general domain, and train the summarization model through a cost function;

[0085] To clarify, the general domain refers to performance on a wide range of language tasks and language data. Training a summarizing model based on the general domain enables the summarizing model to be pre-trained on a large-scale general corpus, which facilitates more comprehensive performance on various natural language processing tasks and ensures the performance of the summarizing model.

[0086] The cost function can be used to measure the difference between the model's predicted results and the actual labels during the training process. Preferably, in this embodiment of the invention, maximum likelihood estimation is used as the cost function. It can be noted that maximum likelihood estimation (MLE) is a commonly used statistical method that can maximize the probability of model parameters given training data.

[0087] S32: Sample the output of the summarization model and obtain multiple initial summaries with different generation probabilities for the same input through the beam search algorithm;

[0088] As explained, beam search is a search algorithm used in natural language processing tasks to generate better sequence outputs. Beam search considers a series of possible output sequences over multiple time steps and retains a certain number of candidate sequences. This algorithm allows the model to consider longer-term contextual information when generating sequences, which helps to reduce generation errors and improve the quality of the generated results.

[0089] It can be explained that the beam search algorithm generates multiple initial summaries with different generation probabilities for the same input. In other words, generating multiple initial summaries through the beam search algorithm facilitates subsequent calculations, and calculating the generation probability of the initial summaries is to determine which of the multiple initial summaries is reasonable.

[0090] S33: Score the initial summary based on the scoring model;

[0091] Understandably, a scoring model is used to score multiple initial summaries. The scoring model is a model used to evaluate the quality of each initial summary, that is, to combine generation probability and scoring to obtain generation results that are more consistent with the facts.

[0092] Understandably, a high-quality initial summary can be obtained through a pre-trained summarization model, while using a beam search algorithm can generate multiple initial summaries, increasing the diversity of summaries, covering more summary content and styles, and providing more comprehensive information; further calculating the generation probability of the initial summaries allows for the evaluation of the rationality and accuracy of each initial summary.

[0093] Furthermore, the cost function for training the summarizing model is:

[0094]

[0095] in, Indicates the first i The probability of generating an initial summary.

[0096] Furthermore, the initial summaries are sorted based on the generation probability and the scoring, and the specific formula for calculating the loss function based on the sorting result is as follows:

[0097]

[0098]

[0099] in, Indicates the first i Scoring of the initial summary; Indicates the first j Scoring of the initial summary; Indicates the first i The probability of generating an initial summary; Indicates the first j The probability of generating an initial summary; Indicates satisfaction , Indicates dissatisfaction .

[0100] Please combine Figure 1 and Figures 4-5 Understandably, using a loss function during the ranking process can measure the quality of the generated initial summaries, making it easier to train the summarization model to obtain initial summaries with higher factual consistency scores; specifically, if the training results satisfy... The score is 1, which does not meet the requirements. If the score is 0, then the score will be 0.

[0101] It is explained that the generation probability and the initial summary after scoring are both sorted in the same way. Specifically, keeping the same sorting means that if the initial summary of a certain output has the highest generation probability, it will also get the highest score after being trained by the scoring model, which makes the final generated summary best satisfy the consistency of facts.

[0102] Understandably, the generation probability and the scored initial summaries are sorted in the same way. That is, in the beam search algorithm, the initial summaries with higher generation probabilities are selected as candidates for the final summaries to be generated. Then, it is determined whether the initial summaries with higher generation probabilities also obtain the highest scores through the scoring model. If so, the summaries generated through this method also have the highest factual consistency. If not, the initial summaries that obtain higher scores during the scoring process are then selected as candidates. Finally, the generation probabilities and the scores of the scored initial summaries are neutralized so that the initial summaries with higher factual consistency scores have a greater generation probability. This sorting method ensures that the final output summary is based on the result of multiple selections, which is conducive to improving the generation quality of the summary.

[0103] It can be explained that during the sorting process, the probability of generating a low-quality initial summary and the score obtained by the scoring model are both relatively low, thus reducing the probability of it being the final generated summary. Furthermore, if both the generation probability and the obtained score are low during the sorting process, the ranking will be lower. If only one of these two factors is sorted, the result is the same as sorting both together. However, this invention comprehensively judges the generation probability and the obtained score, making the training process of the summarizing model more flexible. When new data is input, the summarizing model can better adapt and perform accurate sorting. In addition, in practical applications, a weight is set between the generation probability and the obtained score to influence the sorting result. This setting ensures that the summarizing model pays more attention to important data and is not too affected by poor data. This not only improves the robustness and flexibility of the summarizing model but also allows it to better adapt to sorting needs under different circumstances through reasonable weight settings. Moreover, some erroneous initial summaries generated by the beam search algorithm can be filtered and avoided through the loss function, reducing the error in generating summaries. That is, the loss function can evaluate the quality of different generated results and guide its learning, facilitating the subsequent generation of higher-quality summaries.

[0104] Please see Figure 6 The second embodiment of the present invention provides a sorting-based summary generation training system 1, comprising the following modules:

[0105] Construction module 10 is used to obtain training samples, which include positive samples and negative samples, and to construct training data based on the combination of positive and negative samples;

[0106] Processing module 11 is used to obtain the scoring model, input training data into the scoring model for training, so that the scoring model scores positive samples higher than negative samples.

[0107] The generation module 12 is used to acquire and train the summarization model, sample the output of the summarization model to obtain multiple initial summaries with different generation probabilities for the same input, score the initial summaries based on the scoring model, sort the initial summaries based on the generation probability and the score, calculate the loss function based on the sorting result, and train the summarization model so that the initial summaries with higher fact consistency scores have a higher generation probability.

[0108] Understandably, when the modules of the sorting-based summary generation training system 1 are in operation, they need to utilize a sorting-based summary generation method provided in the first embodiment. Therefore, whether the construction module 10, processing module 11 and generation module 12 are integrated or different hardware is configured to produce functions with similar effects to those achieved by the present invention, they all fall within the protection scope of the present invention.

[0109] Please combine Figures 4-6 Specifically, firstly, the construction module 10 combines positive and negative samples to construct the training data for the entire method. Then, based on the processing module 11, the constructed training data is input into the scoring model for training, enabling the scoring model to score the factual consistency of positive and negative samples. Next, the generation module 12 inputs the training data that satisfies factual consistency into the summary model that has determined the baseline, and samples the summary model to obtain multiple initial summaries with different generation probabilities from the same input. Then, the initial summaries are input into the scoring model to obtain assigned scores, which can summarize the quality or relevance of the initial summaries for subsequent operations. The initial summaries are sorted in descending order by generation probability and obtained score, that is, the initial summaries with higher generation probability and score are ranked higher in the list. The loss function is calculated based on the sorting result, and the summary model is trained so that the initial summaries with higher factual consistency scores have a greater generation probability.

[0110] The third embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed, implements a sorting-based summary generation training method as provided in the first embodiment of the present invention.

[0111] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0112] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to the invention.

[0113] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It is particularly important to note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] Compared with existing technologies, the sorting-based summary generation training method, system, and storage medium provided by this invention have the following beneficial effects:

[0116] 1. A sorting-based summary generation method provided in this embodiment of the invention includes the following steps: First, training samples are obtained, including positive and negative samples, and training data is constructed based on the combination of positive and negative samples; then, a scoring model is obtained, and the training data is input into the scoring model for training, so that the scoring model scores the factual consistency of positive and negative samples; next, a summary model is obtained and trained, and the output of the summary model is sampled to obtain multiple initial summaries with different generation probabilities for the same input, and the initial summaries are scored based on the scoring model; finally, the initial summaries are sorted based on the generation probability and the scored initial summaries, a loss function is calculated based on the sorting result, and the summary model is trained so that the initial summaries with higher factual consistency scores have a higher generation probability; that is, a scoring model is trained by constructing training samples to score the factual consistency of the generated results, and then the output of the summary model is sampled. This method obtains multiple initial summaries with different generation probabilities from the same input, and sorts them according to their generation probabilities and scores to train the summarization model, resulting in an initial summary with a higher factual consistency score. This improves the factual consistency of the generated summaries and ensures the performance of the summarization model. Furthermore, this method has low training costs, does not require additional labeled data to train the scoring model, and its implementation uses both positive and negative samples to train the model more comprehensively, ensuring the quality and accuracy of the generated summaries. Scoring the factual consistency of positive and negative samples helps improve the model's ability to select training samples, resulting in greater factual consistency and more comprehensive, accurate, and targeted generated summaries. Additionally, sorting the initial summaries based on their generation probabilities and scores and then training the summarization model allows for more efficient acquisition of generated summaries that meet factual consistency requirements, and the training stability is good.

[0117] 2. The positive samples provided in this embodiment of the invention include manually labeled sample data that meets the requirements of factual consistency and is collected from existing databases. This setting ensures that the data of the positive samples is based on real facts and can provide a reliable data source. If there are no positive samples from factual sample data during the model training process, the factual consistency of the generated summary will be reduced, resulting in the generation of false information.

[0118] 3. The method for obtaining negative samples provided in this embodiment of the invention specifically includes the following steps: First, the source text is obtained, which includes the source summary and the source text. The source text is labeled based on the word list annotation model, and the entity nouns or verbs in the source text and the source summary are labeled respectively. Then, based on the entity nouns or verbs in the positive samples, at least one entity noun or verb is selected. Finally, based on the labeled entity nouns or verbs, the selected entity nouns or verbs are replaced to generate sample data with different content but the same part of speech. This setting can increase the diversity of data, improve the generalization ability of negative samples by replacing entity nouns and verbs, reduce data bias, and enable the subsequent scoring model to better understand and classify different types of summaries, thereby improving factual consistency. It can also reduce the repetition of samples and avoid overfitting of the model. For generating sample data with different content but the same part of speech, it can help the algorithm learn a wider range of semantic representations, which is beneficial to the generation of summaries and improves the generation quality.

[0119] 4. The scoring model provided in this embodiment of the invention trains the model by inputting training data into the scoring model, enabling the scoring model to score the factual consistency of positive and negative samples. Specifically, the training data consists of sample pairs constructed from documents and summaries. The sample pairs are concatenated and input into the BERT model. Based on the BERT model, the sample pairs are labeled with 0-1, with positive samples labeled as 1 and negative samples labeled as 0. This setting is used to judge whether the training samples meet the factual consistency requirement. It makes full use of the information of the sample pairs. By concatenating the sample pairs constructed from documents and summaries and inputting them into the model, the information of the two can be fused to more comprehensively represent the semantic features of the sample pairs, avoiding the model using single feature information. This helps the model improve the accuracy of distinguishing between positive and negative samples and ensures the performance of the scoring model.

[0120] 5. The present invention provides a method for obtaining and training a summarizing model, sampling the output of the summarizing model to obtain multiple initial summaries with different generation probabilities for the same input, and scoring the initial summaries based on a scoring model. Specifically, this includes the following steps: obtaining a summarizing model based on a general domain and training it using a cost function; sampling the output of the summarizing model and obtaining multiple initial summaries with different generation probabilities for the same input using a beam search algorithm; scoring the initial summaries based on a scoring model; the trained summarizing model can obtain high-quality initial summaries, while the beam search algorithm can generate multiple initial summaries, increasing the diversity of summaries, covering more summary content and styles, and providing more comprehensive information; further calculating the generation probability of the initial summaries allows for the evaluation of the rationality and accuracy of each initial summary.

[0121] 6. The present invention also provides a sorting-based summary generation training system and a computer-readable storage medium, which have the same beneficial effects as the sorting-based summary generation training method described above, and will not be repeated here.

[0122] The foregoing has provided a detailed description of a sorting-based summary generation training method, system, and storage medium disclosed in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention. Any modifications, equivalent substitutions, and improvements made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A sorting-based summary generation training method, characterized in that: Includes the following steps: Obtain training samples, which include positive samples and negative samples, and construct training data based on a combination of positive and negative samples; Obtain the scoring model, input the training data into the scoring model for training, and enable the scoring model to score the factual consistency of positive and negative samples; Obtain and train a summarization model, sample the output of the summarization model to obtain multiple initial summaries with different generation probabilities for the same input, and score the initial summaries based on a scoring model; The initial summaries are sorted based on their generation probability and scores. A loss function is calculated based on the sorting results, and a summarization model is trained so that the initial summaries with higher fact consistency scores have a higher generation probability. The ranking is based on the generation probability and the initial summary after scoring. The specific formula for calculating the loss function based on the ranking result is as follows: in, Indicates the first i Scoring of the initial summary; Indicates the first j Scoring of the initial summary; Indicates the first i The probability of generating an initial summary; Indicates the first j The probability of generating an initial summary; Indicates satisfaction , Indicates dissatisfaction .

2. The sorting-based summary generation training method as described in claim 1, characterized in that: The positive samples include manually labeled sample data that meets factual consistency and are collected from existing databases.

3. The sorting-based summary generation training method as described in claim 2, characterized in that: The method for obtaining negative samples specifically includes the following steps: Obtain the source text, which includes the source summary and the source text. Tag the source text based on the word list annotation model, and tag the entity nouns or verbs in the source text and the source summary respectively. Sampling is performed based on entity nouns or verbs in the positive samples, selecting at least one entity noun or verb; Based on the tagged entity nouns or verbs, the selected entity nouns or verbs are replaced to generate sample data with different content but the same part of speech.

4. The sorting-based summary generation training method as described in claim 1, characterized in that: The scoring model is the BERT model.

5. The sorting-based summary generation training method as described in claim 4, characterized in that: To obtain a scoring model, the training data is input into the scoring model for training, enabling the scoring model to score the factual consistency of positive and negative samples. Specifically, this involves: The training data consists of sample pairs constructed from documents and summaries. The sample pairs are concatenated and input into the BERT model. The sample pairs are labeled 0-1 based on the BERT model, with positive samples labeled as 1 and negative samples labeled as 0.

6. The sorting-based summary generation training method as described in claim 1, characterized in that: The process involves acquiring and training a summarization model, sampling the model's output to obtain multiple initial summaries with different generation probabilities for the same input, and scoring the initial summaries based on a scoring model. The specific steps include: Obtain a summary model based on a general domain and train the summary model using a cost function; The output of the summarization model is sampled, and multiple initial summaries with different generation probabilities for the same input are obtained through a beam search algorithm. The initial summary is scored based on a scoring model.

7. The sorting-based summary generation training method as described in claim 6, characterized in that: The cost function for training the summarization model is: in, Indicates the first i The probability of generating an initial summary.

8. A sorting-based summary generation training system, characterized in that: Includes the following modules: A construction module is used to obtain training samples, which include positive samples and negative samples, and to construct training data based on the combination of positive and negative samples. The processing module is used to obtain the scoring model, input the training data into the scoring model for training, so that the scoring model scores positive samples higher than negative samples. The generation module is used to acquire and train the summarization model, sample the output of the summarization model to obtain multiple initial summaries with different generation probabilities for the same input, score the initial summaries based on the scoring model, sort the initial summaries based on the generation probability and the score, calculate the loss function based on the sorting result, and train the summarization model so that the initial summaries with higher fact consistency scores have a higher generation probability. The ranking is based on the generation probability and the initial summary after scoring. The specific formula for calculating the loss function based on the ranking result is as follows: in, Indicates the first i Scoring of the initial summary; Indicates the first j Scoring of the initial summary; Indicates the first i The probability of generating an initial summary; Indicates the first j The probability of generating an initial summary; Indicates satisfaction , Indicates dissatisfaction .

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, it implements the sorting-based summary generation training method as described in any one of claims 1-7.

Images