Abstract loyalty evaluation method based on data enhancement and two-stage training
By applying data augmentation methods and two-stage training strategies in the text summary data set, the shortcomings of generative text summary fidelity assessment in the prior art are solved, and the system's loyalty assessment ability and negative summary quality are significantly improved.
Patent Information
- Application Number
- CN202411823059.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-03-28
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
It is difficult for the prior art to effectively evaluate the loyalty of generative text summary, especially when constructing unfaithful negative summary, the negative summary generated by the existing method is too different from the original text, and the information from the text summary data set is not fully utilized, affecting the accuracy of the system training effect and faithfulness evaluation.
Three data augmentation methods (entity replacement, similar topic search, extrapolation mask filling) are used to extract training data from the text summary data set, and the system is trained in two stages in multi-task joint. The training data constructed based on the original text and reference abstracts are used to gradually deepen the system's understanding ability.
It significantly improves the system's loyalty evaluation ability, improves the quality of the negative summary generated, enriches the training data set, and enhances the system's training effect and loyalty evaluation ability in complex contexts.
Smart Images
Figure CN119719361A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and relates to a summary fidelity evaluation method based on data enhancement and two-stage training. The method is based on data enhancement and two-stage training mechanism, and aims to evaluate the fidelity of a generated text summary relative to the original text. Background Art
[0002] With the rapid development of the Internet, information has exploded, and text summarization technology has emerged to help users quickly obtain the core content of articles and save time. Text summarization refers to the process of extracting key information from the original text to generate a concise, coherent short text containing the main points. According to the method of summarization generation, text summarization can be divided into extractive and generative methods. Since the generative method is closer to the process of manually generating summaries and has advantages over the extractive method in terms of grammatical accuracy and semantic coherence, it has received more and more attention. In recent years, with the availability of large-scale datasets and the increasing maturity of pre-trained models, research on generative text summarization has made significant progress. However, studies have found that up to 30% of the summaries generated by the generative summary model are inconsistent with the facts of the original text, which seriously hinders the application of automatic text summarization in practical scenarios. Therefore, evaluating the fidelity of text summaries relative to the original text is an urgent problem to be solved.
[0003] Currently, there is no supervised training dataset for fidelity assessment. Manually creating large-scale, high-quality datasets is costly and time-consuming. Therefore, some researchers use weak supervision methods to construct synthetic data through data augmentation to train summary fidelity assessment systems. In text summary datasets, the reference summary is usually considered to be faithful to the original text. Therefore, the main challenge is how to construct an effective unfaithful summary. When constructing unfaithful negative summaries, the existing technologies have the following main limitations:
[0004] First, existing methods for constructing negative summaries are mainly based on simple rule conversion or masked language model tasks. The negative summaries generated based on simple rule conversion are too different from the original text, while the masked language model task easily makes the model overly dependent on common context correspondences, resulting in the masked part being correctly filled. Both cannot generate stable and effective negative summaries, which in turn affects the system training effect.
[0005] Second, the information in the text summary dataset is not fully utilized. Researchers usually construct negative summaries based only on the original text or reference summary, and fail to fully integrate and utilize these two types of information, resulting in the system lacking sufficient information to learn the complex relationship between the original text and the summary during training, which ultimately affects the accuracy of the fidelity assessment. Summary of the invention
[0006] In view of the above problems, the present invention proposes an innovative summary fidelity evaluation system based on data enhancement and two-stage training. First, three data enhancement methods (entity replacement, similar retrieval with the same subject, and external mask filling) are applied to extract training data from the text summary dataset. Entity replacement (ES) replaces the entity in a single sentence with other entities of the same category and different names, which is used as a negative summary. Similar retrieval with the same subject (S3T) groups the articles in the dataset according to the topic, and selects single sentences with similar semantics in the articles with the same topic as unfaithful summaries. External mask filling (IFEM) inserts additional masks into single sentences and fills them to generate negative summaries that are related to the original content but unfaithful. Secondly, the information of the text summary dataset is fully utilized, and the system is trained for multi-task joint training in two stages. The first stage uses training data extracted based on the original text to train the system to master the ability to judge the consistency of basic facts; the second stage uses training data extracted based on reference summaries to improve the system's fidelity evaluation ability in complex contexts.
[0007] The present invention significantly improves the system's fidelity assessment capability by improving the negative summary construction method and making full use of data set information, providing strong technical support for the practical application of automatic text summarization. This method can, to a certain extent, solve the shortcomings of existing fidelity assessment methods in utilizing text summary data sets, as well as the problem that the constructed negative summary is significantly different from the original text, thereby improving the system's fidelity assessment capability.
[0008] To achieve the above object, the present invention provides the following solutions:
[0009] S1, extract single sentences from the original text and reference summaries of the text summarization dataset;
[0010] S2, construct training data based on the extracted single sentences using three data augmentation methods;
[0011] S3, using the training data constructed based on the original text to train the system in the first stage;
[0012] S4, using the training data constructed based on the reference summary to train the system in the second stage;
[0013] S5. The user provides a candidate summary and the corresponding original text, and the system determines whether the summary is faithful.
[0014] Furthermore, the S1 specifically includes:
[0015] Extract single sentences from source and reference summaries of a text summarization dataset.
[0016] First, the original text and reference abstracts are divided into sentences according to certain rules; then, sentences that meet the length requirements are extracted from the front to the back. Since most articles adopt a general-specific or general-specific-general structure, the front sentences contain key factual information. This extraction method helps to extract key facts and lays the foundation for building high-quality negative abstracts.
[0017] The beneficial effect of adopting the above further scheme is that by determining reasonable sentence segmentation rules and extracting single sentences from the front to the back, the factual information of the original text and the reference summary can be effectively extracted, providing a basis for the subsequent construction of a high-quality unfaithful summary.
[0018] Furthermore, the S2 specifically includes:
[0019] Three data augmentation methods are applied based on the extracted single sentences to construct training data.
[0020] The data enhancement methods include entity replacement (ES), similar retrieval with the same subject (S3T), and interpolation mask filling (IFEM). For a target sentence, the ES method replaces the entity in the sentence with other entities of the same category but with different names; the S3T method groups the articles in the dataset according to the topic, and selects the sentences with similar semantics but different facts from the target sentence in the articles with the same topic as the negative summary; the IFEM method inserts an additional mask into the target sentence and fills it in, generating a negative summary that is related to the original content but not faithful.
[0021] The beneficial effect of adopting the above further scheme is that by applying three data augmentation methods, negative summaries that are highly relevant but not faithful to the original content can be generated, which ensures the quality of the negative summary and helps to improve the training effect and evaluation performance of the system.
[0022] Furthermore, the S3 specifically includes:
[0023] The system is trained in the first phase using training data constructed based on the original text.
[0024] The system is based on the BERT model and uses a fully connected layer as a classifier. Specifically, the summary and the original text are first concatenated as the input of the BERT model; then the vector corresponding to the "[CLS]" tag in the output of the BERT model is extracted as the representation of the entire input sequence, and the vector is simultaneously input into the binary and multi-classification classifiers for prediction and classification to achieve multi-task joint training. Among them, the binary classification task aims to train the system to distinguish between faithful summaries and unfaithful summaries; the multi-classification task aims to train the system to distinguish between the three types of unfaithful summaries and faithful summaries generated by the ES, S3T, and IFEM methods. If the system can effectively distinguish these categories, it means that it can keenly capture the subtle differences between summaries, which is crucial for binary classification tasks.
[0025] The beneficial effect of adopting the above further scheme is that through the first stage of training, the system can accurately identify direct and clear factual correspondences, keenly capture basic consistency and differences, and lay the foundation for the system's overall evaluation and classification capabilities.
[0026] Furthermore, the S4 specifically includes:
[0027] The system is trained in the second phase using training data constructed based on the reference summaries.
[0028] The system trained in the first phase is used as the basis for the second phase of training. This phase uses training data constructed based on single sentences of reference summaries, and the training method is the same as the first phase.
[0029] The beneficial effect of adopting the above further scheme is: through the second stage of training, the system is guided to master a deeper level of consistency judgment logic, requiring the system to maintain a high level of fidelity assessment ability in a generalized context.
[0030] Furthermore, the S5 specifically includes:
[0031] The user provides a candidate summary and the corresponding original text, and the system determines whether the summary is faithful.
[0032] First, the candidate summary and the original text are concatenated as the input of the BERT model. Then, based on the training in the above two stages, the BERT model comprehensively understands the contextual information of the input sequence and outputs a "[CLS]" vector representing the entire input sequence. Finally, the vector is input into a binary classifier for prediction to determine whether the candidate summary is faithful to the original text.
[0033] The beneficial effect of adopting the above further scheme is that through two stages of progressive training, the system can deeply understand the potential factual correspondence between the candidate summary and the original text, thereby improving the accuracy of the fidelity assessment.
[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0035] The present invention defines multiple data enhancement methods to generate negative summaries that are related to the original content but not faithful, effectively improving the quality of negative summaries, enriching the training data set, and significantly improving the training effect of the system.
[0036] Based on the training data constructed from the original text and reference summaries, the present invention conducts multi-task joint training on the system in two stages, making full use of the information of the text summary dataset, while gradually deepening the system's understanding ability and improving the system's fidelity assessment ability in complex contexts. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A framework diagram of the summary fidelity evaluation system provided by an embodiment of the present invention, showing the architecture of the entire evaluation system and the relationship between the modules;
[0038] Figure 2 An example diagram of the ES method provided by an embodiment of the present invention;
[0039] Figure 3 An example diagram of the S3T method provided in an embodiment of the present invention, showing the operation flow of the method and an example of a negative summary generated;
[0040] Figure 4 An example diagram of the IFEM method provided in an embodiment of the present invention shows the operation flow of the method and an example of a negative summary generated. DETAILED DESCRIPTION
[0041] The implementation process of the present invention will be described in detail below in conjunction with the accompanying drawings and technical solutions.
[0042] Step 1: Extract single sentences from the original text and reference summaries of the text summarization dataset.
[0043] (1) Sentence division principle: The original text and reference abstracts are divided into three levels. The first level is divided into sentences with sentence-level labels (".", "!"", "?"", "!"", "?""); if the sentence length exceeds 200 words, the second level is divided into sentences with "", ":", "-", ";", ":", ";"; if the second level of sentence division is still more than 200 words, the third level is directly sliced according to the length of 200.
[0044] (2) Extraction method: Extract sentences that meet the length requirement (length ≥ 15) from the front to the back. Since most articles adopt a general-specific or general-specific-general structure, the front sentences often contain key factual information. This extraction method helps to extract key facts and lays the foundation for constructing high-quality negative summaries.
[0045] Step 2: Apply three data augmentation methods based on the extracted single sentences to construct training data.
[0046] (1) Data augmentation methods
[0047] ① Entity Substitution (ES) method: Replace the entities in a single sentence with other entities of the same category but different names. For example, "[Beijing, LOC]" can be replaced with "[Shanghai, LOC]", but cannot be replaced with "[China, ORG]". The ES method preferentially performs substitutions within the sentence to maintain the highest possible relevance to the original sentence content. Specifically, entities in a single sentence adopt a three-level substitution strategy: If the number of entities of a certain category in a single sentence exceeds 1, the entities of that category are swapped with each other; if the number of entities is insufficient, the substitution scope is expanded to entities of the same category in the original text; if there are still remaining entities, the substitution scope is further expanded to entities of the same category in the entire dataset. This hierarchical substitution strategy ensures a high degree of relevance to the original text while introducing factual errors. Figure 2 Shows an example of the ES method.
[0048] ② Similar Sentence Retrieval for the Same Topic (S3T) method: Aims to find a single sentence that is semantically highly similar but not exactly the same for each single sentence, and use it as a negative summary.
[0049] First, group the articles in the dataset according to the topic. The present invention uses the LDA topic model to mine the potential topics in the dataset. Determining the optimal number of topics is crucial for improving the accuracy of topic clustering. To this end, the present invention trains multiple LDA models on the dataset with different numbers of topics, and compares the performance of different models through the topic coherence index to determine the optimal number of topics, so as to group the articles by topic.
[0050] Then, for the target single sentence, use BERTScore to retrieve semantically similar single sentences in the articles of the same topic (excluding the article to which the target single sentence belongs), and select the single sentence with the highest F1 score as the negative summary. Through this method, it is possible to identify the single sentence that is semantically closest to the target single sentence, while ensuring that these single sentences are not exactly equivalent to the target single sentence in terms of factual description. Figure 3 Shows an example of the S3T method.
[0051] ③ Interpolation Mask Filling (IFEM) method: Insert additional masks while preserving the original content of the target single sentence, guiding the model to generate a negative summary that is highly relevant to the original text but contains disloyal information.
[0052] Figure 4 Shows an example of the IFEM method. First, perform lexical analysis on the target single sentence, and selectively insert masks before specific parts of speech (nouns, abbreviations, numerals, quantifiers, quantity words, time words, locative words, place words, verbs, adjectives). Based on the length distribution of the single sentence, randomly select m words. For example Figure 4As shown in the figure, the bold part in the masked sentence represents the selected words, and the mask is inserted before these words. Then, the mask is filled using the pre-trained masked language model. The sampling algorithm uses Top-K, that is, k words with the highest probability are retained as candidates at each step. For each input, the model returns k outputs, calculates the ROUGE-1 score of each output and the target sentence, and finally selects the one with the lowest score as the result. Figure 4 In the generated sentences shown, the underlined part indicates the content filled by the model, and the bold part indicates the unfaithful content. This work uses the THULAC Chinese lexical analysis tool for part-of-speech tagging and the pre-trained model BART-base for mask filling.
[0053] (2) Construction process
[0054] Taking the reference summary as an example, a single sentence sj is extracted, where j∈[1,3], and the above three methods are applied to generate negative summaries. For sj, where j=4, as a positive summary, the original text is processed in the same way.
[0055] Step 3: Use the training data constructed based on the original text to perform the first phase of training on the system.
[0056] (1) Training basis: Based on the BERT model, the fully connected layer is used as the classifier.
[0057] (2) Training data: The training data is constructed based on the original single sentence using data augmentation method. It has a low level of abstraction and obvious factual errors.
[0058] (3) Training method: The concatenated summary and the original text (connected with "[SEP]", and "[CLS]" is added before the summary) are used as the input of the BERT model, and the vector corresponding to the "[CLS]" tag in the output of the BERT model is extracted and input into the binary and multi-classification classifiers to predict the classification, thus realizing multi-task joint training. The binary classification task training system distinguishes between faithful and unfaithful summaries, and the multi-classification task training system distinguishes between the three types of unfaithful summaries and faithful summaries generated by the ES, S3T, and IFEM methods. Effectively distinguishing these categories is helpful for the binary classification task. The classification task uses cross entropy as the loss function. The loss functions of the binary classification task and the multi-classification task are shown in formulas (1) and (2).
[0059]
[0060] Among them, yi represents the actual label of the i-th sample, represents the probability that the i-th sample is predicted to be faithful, yi,c is the c-th element in the one-hot encoding of the actual label of the i-th sample, is the probability that the i-th sample is predicted to be the c-th category.
[0061] The total training loss L consists of the binary classification loss Ltwo and the multi-classification loss Lmulti, and the calculation formula is shown in formula (3).
[0062] L=αLtwo+(1-α)Lmulti(3)
[0063] Among them, α represents the loss weight. Through reasonable weight distribution, the system can achieve an effective learning balance between the two tasks and further improve the accuracy of the summary fidelity evaluation task.
[0064] Step 4: Perform the second phase of training on the system using training data constructed based on the reference summaries.
[0065] (1) Training basis: based on the system after the first stage of training.
[0066] (2) Training data: The training data is constructed based on the reference summary sentence through data augmentation method. It has a high degree of abstraction and a complex correspondence with the original facts.
[0067] (3) Training method: Same as the first stage, it aims to guide the system to master a deeper level of consistency judgment logic so that it can maintain high fidelity evaluation capabilities in a generalized context.
[0068] Step 5: The user provides a candidate summary and the corresponding original text, and the system determines whether the summary is faithful.
[0069] (1) Input processing: The user provides a candidate summary and the original text, which are concatenated (connected with "[SEP]" and "[CLS]" is added before the summary) as the input of the BERT model.
[0070] (2) Evaluation process: The BERT model captures contextual information. The vector corresponding to the “[CLS]” position in its output can be used as the representation of the entire input sequence. This vector is input into a binary classifier for prediction to determine whether the candidate summary is faithful to the original text.
[0071] The implementation method of the present invention is described in detail above, but the present invention is not limited to the above implementation method. The three data enhancement methods effectively improve the quality of negative summaries, can generate summaries that are related to the original text but not faithful, enrich the training data set, and improve the training effect of the system. The two-stage training strategy makes full use of the information of the text summary data set, constructs training data based on the original text and reference summaries, and conducts training in stages, gradually deepening the system's understanding ability and improving the fidelity assessment ability in complex contexts.
Claims
1. A summary fidelity evaluation method based on data enhancement and two-stage training, characterized in that Here are the steps: S1, extract single sentences from the original text and reference summaries of the text summarization dataset; S2, construct training data based on the extracted single sentences using three data augmentation methods; S3, using the training data constructed based on the original text to train the system in the first stage; S4, using the training data constructed based on the reference summary to train the system in the second stage; S5. The user provides a candidate summary and the corresponding original text, and the system determines whether the summary is faithful.
2. According to claim 1, a summary fidelity evaluation method based on data enhancement and two-stage training is characterized in that: The S1 specifically includes: Extract single sentences from the original text and reference summaries of the text summary dataset; first, segment the original text and reference summaries according to certain rules; then, extract sentences that meet the length requirements from front to back; since most articles adopt a general-specific or general-specific-general structure, the front sentence contains key factual information, and this extraction method helps to extract key facts, laying the foundation for building high-quality negative summaries.
3. According to claim 1, a summary fidelity evaluation method based on data enhancement and two-stage training is characterized in that: The S2 specifically includes: Three data augmentation methods are applied based on the extracted single sentences to construct training data; The data enhancement methods include entity replacement (ES) method, same-topic similarity retrieval (S3T) method and interpolation mask filling (IFEM) method; for the target sentence, the ES method replaces the entity in the sentence with other entities of the same category but different names; the S3T method groups the articles in the dataset according to the topic, and selects the sentences with similar semantics but factual differences from the target sentence in the articles with the same topic as the negative summary; the IFEM method inserts an additional mask into the target sentence and fills it in, generating a negative summary that is related to the original content but not faithful.
4. According to claim 1, a summary fidelity evaluation method based on data enhancement and two-stage training is characterized in that: The S3 specifically includes: using training data constructed based on the original text to perform the first phase of training on the system; The system is based on the BERT model and uses a fully connected layer as a classifier. Specifically, the summary and the original text are first concatenated as the input of the BERT model. Then, the vector corresponding to the "[CLS]" tag in the output of the BERT model is extracted as the representation of the entire input sequence. The vector is simultaneously input into the binary and multi-classification classifiers for prediction and classification, thereby achieving multi-task joint training. The binary classification task aims to train the system to distinguish between faithful summaries and unfaithful summaries. The multi-classification task aims to train the system to distinguish between three types of unfaithful summaries and faithful summaries generated by the ES, S3T, and IFEM methods. If the system can effectively distinguish these categories, it means that it can keenly capture the subtle differences between summaries, which is crucial for binary classification tasks.
5. According to claim 1, a summary fidelity evaluation method based on data enhancement and two-stage training is characterized in that: The S4 specifically includes: The system is trained in the second phase using training data constructed based on the reference summaries; The system trained in the first stage is used as the basis for the second stage training; this stage uses training data constructed based on reference summary single sentences, and the training method is the same as the first stage.
6. The summary fidelity evaluation method based on data enhancement and two-stage training according to claim 1, characterized in that: S5 specifically includes: the user provides a candidate summary and the corresponding original text, and the system determines whether the summary is faithful; first, the candidate summary and the original text are concatenated as input to the BERT model; then, based on the above two stages of training, the BERT model comprehensively understands the contextual information of the input sequence and outputs a "[CLS]" vector representing the entire input sequence; finally, the vector is input into a binary classifier for prediction to determine whether the candidate summary is faithful to the original text.