Optimization method for internal illusion problem of generative text abstract model

Abstract

The invention relates to the technical field of natural language processing, in particular to a generative text abstract model internal illusion problem optimization method, which comprises the following steps: acquiring a generative text abstract model and source text data; iteratively updating the model to obtain a first text abstract model and candidate abstracts; calculating multi-granularity quality features of the candidate abstracts, and training the first text abstract model to obtain a second text abstract model; identifying and verifying key entities of the source text data and the candidate abstracts to obtain content loyalty representation; fusing the candidate abstracts, the multi-granularity quality features and the content loyalty representation to obtain optimized abstract generation features; processing the optimized abstract generation features by adopting a second text abstract model to obtain an abstract text for optimizing the illusion phenomenon; the method can effectively reduce the illusion phenomenon commonly existing in the generative text abstract model, and has the characteristic of high universality.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for optimizing the intrinsic hallucination problem of a generative text summarization model. Background Art

[0002] Generative text summarization models can automatically generate concise and accurate summaries from original text. The hallucination problem occurs when generative text summarization models mistakenly generate information that is not present in the original text. This typically manifests as a summary that includes facts, data, or details not mentioned in the original text.

[0003] There are two main ways to solve the hallucination problem: the method of training additional discriminant models: by introducing an additionally trained discriminant model to evaluate the factual consistency between the generated summary and the source document; the method based on prompt words: by inputting specific instructions into the pre-trained language model to guide the model to generate text output that meets the expectations.

[0004] The existing technology has the following problems: The method of training additional discriminant models: It does not optimize and improve the hallucination problem in the model from the perspective of the original summary model training method. It requires training additional discriminant models, increases resource consumption, and cannot guarantee the performance of the discriminant model; the method based on prompt words: It relies heavily on the quality of the prompt words. If the quality of the generated prompt words is poor, the quality of the generated text corresponding to the model will be severely reduced, and may even be worse than the original effect. In addition, it is difficult to design prompt words that are effective for all data.

[0005] Therefore, there is an urgent need for a method that can effectively solve the inherent hallucination problem of generative text summarization models. Summary of the Invention

[0006] In view of this, the present invention discloses an optimization method for the inherent hallucination problem of a generative text summarization model to solve the above problem, including:

[0007] S1. Obtain a generative text summarization model with hallucination problems and source text data;

[0008] S2. Using the source text data, iteratively update the generative text summarization model with the hallucination problem to obtain a first text summarization model and candidate summaries;

[0009] S3. Calculate the semantic evaluation index of the candidate summary, use the semantic evaluation index as the multi-granularity quality feature of the candidate summary, and use the multi-granularity quality feature to train the first text summary model to obtain the second text summary model;

[0010] S4. Use named entity recognition technology to identify key entities in the source text data and the candidate summary, and use the key entities in the source text data to verify the key entities in the candidate summary to obtain a content fidelity representation;

[0011] S5. Fusion of candidate summaries, multi-granularity quality features, and content fidelity representation to obtain optimized summary generation features.

[0012] S6. Use the second text summary model to process the optimized summary generation features to obtain a summary text that optimizes the hallucination phenomenon.

[0013] The beneficial effects of the present invention include:

[0014] By conducting a multi-dimensional evaluation of the BARTScore, semantic consistency, and relevance of the initially generated summaries, the quality of candidate summaries is assessed, the most representative semantic fragments in the final summary are identified and highlighted, and the generation paths of core elements such as specific entities or key factual statements are analyzed. This helps reduce the occurrence of hallucination problems at the data level during the candidate summary generation stage.

[0015] By designing label smoothing to learn summary generation, we optimize the generated sequence at the token level, analyze summary information from a more fine-grained perspective, and reduce the occurrence of hallucination problems during the model training phase. This further alleviates the inherent hallucination problem of generative text summarization models and enables this method to have broader natural language processing applications.

[0016] By performing post-summary generation processing, named entity recognition technology and replacement algorithm are used to ensure that the key entities in the summary are faithful to the source text, thereby reducing the occurrence of hallucination problems from the post-summary generation processing stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process of optimizing the intrinsic hallucination problem of the generative text summarization model in the present invention;

[0018] Figure 2 A schematic diagram of the process of identifying key entities in the present invention;

[0019] Figure 3 A schematic diagram showing the calculation and visualization of the information alignment score in the present invention;

[0020] Figure 4 This is an example diagram of candidate summaries generated by a model using the method of the present invention in an embodiment of the present invention;

[0021] Figure 5 1. A comparison chart of novelty scores of a model using the method of the present invention and a baseline model in an embodiment of the present invention;

[0022] Figure 6This is an example diagram of the generation summary of the model using the method of the present invention and the baseline model in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions, features and advantages of the present invention more clearly understood, the present invention is further described below with reference to the accompanying drawings and embodiments.

[0024] This embodiment includes an optimization method for the intrinsic hallucination problem of a generative text summarization model, specifically, Figure 1 Shown, including:

[0025] S1. Obtain a generative text summarization model with hallucination problems and source text data.

[0026] Specifically, the source text data includes the source text and the reference abstract.

[0027] S2. Using the source text data, iteratively update the generative text summarization model with the hallucination problem to obtain a first text summary model and candidate summaries.

[0028] Specifically, iterative updates include:

[0029] S21. Use a generative text summarization model to generate initial candidate summaries based on the source text data.

[0030] S22. Calculate the recall score of the initial candidate summary for the source text data.

[0031] S23: Set a recall score threshold, and select initial candidate summaries with a recall score not lower than the recall score threshold as valid candidate summaries. The recall score threshold is preferably 40.

[0032] S24. Construct an enhanced training dataset, where the enhanced training dataset includes valid candidate summaries and source text data.

[0033] S25. Use the enhanced training data set to train the generative text summarization model and update the generative text summarization model.

[0034] The ROUGE score of the generative text summarization model is calculated in each loop from S21 to S25 until the ROUGE score stabilizes. The first text summarization model is obtained, and the valid candidate summaries from the last loop are used as candidate summaries. In each loop, the initial candidate summaries, valid candidate summaries, and the enhanced training dataset are all updated.

[0035] S3. Calculate the semantic evaluation index of the candidate summary, use the semantic evaluation index as the multi-granularity quality feature of the candidate summary, and use the multi-granularity quality feature to train the first text summary model to obtain the second text summary model.

[0036] Specifically, semantic evaluation indicators include BARTScore, semantic consistency score, and relevance score. The formula used to calculate BARTScore is:

[0037]

[0038] Where m represents the number of candidate summaries, ω t represents the probability distribution of the tth candidate generated text, y t Indicates the current predicted position, y <t Indicates the position that has been predicted before t.

[0039] The formula used to calculate the semantic consistency score is:

[0040] consistency(y,x)=mean(align(y→x))

[0041] align(a→b)=<α1,α2,…,α n >

[0042] Where consistency represents the consistency score, y represents the candidate summary, x represents the source text data, align(a→b) represents the information alignment score from text a to text b, mean represents the mean of the alignment scores for each token, and α i ∈[0,1] represents the confidence that the i-th token in text a is received by text b.

[0043] Specifically, if Figure 3 Shown is a schematic visualization of the calculation of the information alignment score.

[0044] The formula used to calculate the relevance score is:

[0045] relevance(y,x,r)=mean(align(r→y))×mean(align(y→x))

[0046] Where relevance represents the relevance score, r represents the reference summary, align(r→y) represents the information alignment score between the reference summary r and the summary text y, and align(y→x) represents the information alignment score between the summary y and the source text x.

[0047] Furthermore, the first text summarization model is trained using multi-granularity quality features based on the label smoothing method. The formula for smoothing labels is:

[0048]

[0049] in, represents the smoothed label, y hot represents the label represented by the one-hot vector, α represents the weight parameter with a value of 0.1, and k represents the total length of the sentence represented by the one-hot vector.

[0050] Furthermore, the loss function formula adopted is:

[0051]

[0052] Among them, L MLE represents the total loss function, represents the smoothed label of the first text summarization model at position j, represents the generation probability distribution, S represents the reference summary, l represents the total length of the reference summary, D represents the source text, represents the summary of the first j tokens generated, and θ represents the probability distribution of the token to be generated.

[0053] The iterative updating and training approach proposed in this paper assigns the model distinct roles in two phases: first, the summary model acts as an evaluator, assessing the quality of candidate summaries; second, it acts as a summary generator, using the optimal candidate summaries for summary learning and final summary generation. This dual-perspective optimization framework clearly demonstrates the decision-making process in summary generation, specifically how the model ensures the accuracy and completeness of key information.

[0054] S4. Use named entity recognition technology to identify key entities in the source text data and the candidate summary, and use the key entities in the source text data to verify the key entities in the candidate summary to obtain a content fidelity representation. Obtaining a content fidelity representation specifically includes:

[0055] S41, such as Figure 2 As shown in the figure, the BERT model and the bidirectional LSTM neural network are used in sequence to encode the data to be recognized and obtain a context information vector; the data to be recognized is the source text data or the candidate summary.

[0056] Specifically, the bidirectional LSTM neural network includes three parallel bidirectional LSTM neural networks, which are respectively used to extract word pair information, distance information and region information; the context information vector is composed of the word pair information, distance information and region information.

[0057] S42. Perform conditional layer normalization processing on the context information vector to obtain a word pair information matrix, a distance information matrix, and a region information matrix.

[0058] S43. Use dilated convolution to extract features from the word pair information matrix, the distance information matrix, and the region information matrix to obtain word pair information features, distance information features, and region information features.

[0059] S43. Concatenate the word pair information features, the distance information features, and the region information features to obtain the word pair grid features.

[0060] S44. Use the Deep Biaffine algorithm and Multilayer Perceptron (MLP) to calculate the relationship score between the word and the grid feature. According to the relationship score, filter the word and the grid feature to obtain the entity set.

[0061] Specifically, a dynamic threshold τ = μ + βσ is set, where μ represents the mean score of the relationship score, σ represents the standard deviation of the relationship score, and β represents the parameter that controls the degree to which the threshold deviates from the mean. The value of β ranges from -1 to 1.5, preferably 1.2. β determines the strictness of entity screening, and the specific value is determined according to the actual screening requirements. Elements with a score higher than τ in the word pair grid feature are screened to obtain the entity set.

[0062] S45. Calculate the combined feature weights of the entity set based on TF-IDF (Term Frequency–Inverse Document Frequency), sort the elements in the entity set according to the combined feature weights, and select the top five entities with the highest combined feature weights as key entities.

[0063] Specifically, the combined feature weights of the entity set calculated based on TF-IDF include:

[0064] Calculate key factors The TF-IDF statistical method is used to calculate the importance of words in documents. Used to effectively distinguish high-frequency but low-information words.

[0065] Calculating word frequency factors That is, the frequency of occurrence of each entity in the entity set, Used to measure its significance.

[0066] Calculating similarity factor and The word2vec model is used to encode each entity, and the encoding is used as a node in the vector space to calculate the similarity between entities. and the similarity between entities and sentences The sentences are the data to be recognized, that is, the source text data or the candidate summary.

[0067] Further, calculate the combined feature weight CF i , the formula used is:

[0068]

[0069] Among them, CF i represents the combined weight of the i-th key entity.

[0070] S46. Use the key entities of the source text data to verify the key entities of the candidate summary to obtain a content fidelity representation.

[0071] Specifically, the key entities of the source text data are compared with the key entities of the candidate summary; the key entities of the candidate summary are replaced with the key entities of the source text data, the key entity verification is completed, and the obtained candidate summary is used as the content fidelity representation.

[0072] S5. Fusion the candidate summary, multi-granularity quality features, and content fidelity representation to obtain the optimized summary generation feature. The optimized summary generation feature is a one-dimensional vector.

[0073] S6. Use the second text summary model to process the optimized summary generation features to obtain a summary text that optimizes the hallucination phenomenon.

[0074] Furthermore, in comparative tests of various generative text summarization models such as BART, Pegasus and T5, the present invention has the best effect on the BART summary model with hallucination problems, that is, the optimization method of the present invention has the best effect on eliminating hallucinations of the BART summary model.

[0075] Furthermore, since hallucination problems in text summarization often focus on specific entities or key factual statements, the present invention eliminates hallucination phenomena by performing a multi-dimensional evaluation of the BARTScore, semantic consistency, and relevance of the summaries generated by the generative text summarization model, focusing on analyzing the generation paths of core elements such as specific entities or key factual statements.

[0076] Furthermore, this embodiment uses the CNNDM public dataset and the XSum public dataset for testing.

[0077] The CNNDM dataset is a large-scale news dataset commonly used in text summarization tasks. It contains 287,113 training samples and 13,368 test samples. It uses news text as the source text and relevant highlights as the reference summary. Both the source text and the reference summary are relatively long. The XSum dataset is a dataset of highly abstract articles from the British Broadcasting Corporation. It contains 204,045 training samples and 11,332 test samples. Both the source text and the reference summary are relatively short.

[0078] like Figure 4 The following table shows examples of candidate summaries obtained based on the CNNDM public dataset and the XSum public dataset.

[0079] like Figure 5 As shown, using Novelty as a reference indicator and comparing the standard BART model as a baseline model, the present invention achieves the best novelty scores on both datasets and is significantly improved over the baseline model.

[0080] In order to more intuitively demonstrate the quality of the summaries generated by our model, 100 data points were randomly selected from each of the two datasets for spot checking. Figure 6 In the example, the red marks represent the generated text with hallucination problems, while the green marks represent the understanding of the source text semantics and the creative generation of new expressions, which are more in line with the task requirements of generative text summarization. Figure 6 As can be seen from the example, the summary generated by the present invention can effectively reduce the inherent hallucination problem in the summary, because the present invention optimizes the generation sequence at the token level by introducing label smoothing technology, so that the BART model provides an in-depth understanding of text information by analyzing summary information from a more fine-grained perspective. This ability helps to more deeply alleviate the inherent hallucination problem of the generative text summary model and has a wide range of natural language processing application capabilities.

[0081] Finally, it should be noted that the above only describes some embodiments of the present invention. For those skilled in the art, it is conceivable that various changes, modifications, substitutions and deformations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents, and the above-mentioned actions should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing the intrinsic hallucination problem of a generative text summarization model, characterized by: include: S1. Obtain a generative text summarization model with hallucination problems and source text data; S2. Using the source text data, iteratively update the generative text summarization model with the hallucination problem to obtain a first text summarization model and candidate summaries; S3. Calculate the semantic evaluation of the candidate summary, use the semantic evaluation index as the multi-granularity quality feature of the candidate summary, and use the multi-granularity quality feature to train the first text summary model to obtain the second text summary model; the semantic evaluation index includes BARTScore score, semantic consistency score, and relevance score; S4. Use named entity recognition technology to identify key entities in the source text data and the candidate summary, and use the key entities in the source text data to verify the key entities in the candidate summary to obtain a content fidelity representation; S5. Fusion of candidate summaries, multi-granularity quality features, and content fidelity representation to obtain optimized summary generation features. S6. Use the second text summary model to process the optimized summary generation features to obtain a summary text that optimizes the hallucination phenomenon.

2. The optimization method for the intrinsic hallucination problem of the generative text summarization model according to claim 1 is characterized in that Iterative updates to a generative text summarization model with hallucination problems include: S21, using a generative text summarization model to generate initial candidate summaries based on the source text data; S22, calculating the recall score of the initial candidate summary for the source text data; S23. Setting a recall rate score threshold, screening out initial candidate summaries with a recall rate score not lower than the recall rate score threshold as valid candidate summaries; S24. Constructing an enhanced training dataset, where the enhanced training dataset includes valid candidate summaries and source text data; S25. Using the enhanced training data set to train the generative text summarization model, and updating the generative text summarization model; Loop S21 to S25, calculate the ROUGE score of the generative text summarization model in each loop until the ROUGE score stabilizes and the loop ends, obtaining the first text summarization model, and adopting the valid candidate summary in the last loop as the candidate summary.

3. The optimization method for the intrinsic hallucination problem of the generative text summarization model according to claim 2 is characterized in that The recall score threshold is 40.

4. The optimization method for the intrinsic hallucination problem of the generative text summarization model according to claim 1 is characterized in that The first text summarization model is trained using multi-granularity quality features and based on the label smoothing method. The formula for smoothing labels is: in, represents the smoothed label, y hot represents the one-hot vector of the label, α represents the weight parameter with a value of 0.1, and k represents the total length of the sentence represented by the one-hot vector.

5. The optimization method for the intrinsic hallucination problem of the generative text summarization model according to claim 4 is characterized in that: The training is based on the label smoothing method, and the loss function formula used is: Among them, L MLE represents the total loss function, represents the smoothed label of the first text summarization model at position j, Represents the generation probability distribution; the source text data consists of the source text D and the reference summary S, l represents the total length of the reference summary, represents the summary of the first j tokens generated, and θ represents the probability distribution of the token to be generated.

6. The optimization method for the intrinsic hallucination problem of the generative text summarization model according to claim 1, characterized in that Key entities for identifying source text data and candidate summaries include: S41. Use the BERT model and the bidirectional LSTM neural network in sequence to encode the data to be identified to obtain a context information vector; the data to be identified is the source text data or the candidate summary; S42, performing conditional layer normalization processing on the context information vector to obtain a word pair information matrix, a distance information matrix, and a region information matrix; S43, using dilated convolution to extract features from the word pair information matrix, the distance information matrix, and the region information matrix to obtain word pair information features, distance information features, and region information features; S43, concatenating the word pair information features, the distance information features, and the region information features to obtain a word pair grid feature; S44, using a bi-affine named entity recognition algorithm and a multi-layer perceptron to calculate the relationship score between the word and the grid feature, and filtering the word and grid features according to the relationship score to obtain an entity set; S45. Calculate the combined feature weights of the entity set based on TF-IDF, sort the elements in the entity set according to the combined feature weights, and select the top five entities with the highest combined feature weights as key entities.

7. The method for optimizing the intrinsic hallucination problem of a generative text summarization model according to claim 6, characterized in that: The bidirectional LSTM neural network includes three parallel bidirectional LSTM neural networks, which are respectively used to extract word pair information, distance information and region information; the context information vector is composed of the word pair information, distance information and region information.

8. The method for optimizing the intrinsic hallucination problem of a generative text summarization model according to claim 6, characterized in that: Calculating the combined feature weights of an entity set includes: Step 1: Use TF-IDF statistical method to calculate the importance of words in the document Step 2: Calculate the frequency of occurrence of each entity in the entity set Step 3: Use the word2vec model to encode each entity, use the encoding as a node in the vector space, and calculate the similarity between entities respectively. Similarity between the entity and the data to be identified Step 4: Calculate the combined feature weight CF i , the formula used is: Among them, CF i represents the combined weight of the i-th key entity.

9. The method for optimizing the intrinsic hallucination problem of a generative text summarization model according to claim 1, characterized in that: Using the key entities of the source text data to verify the key entities of the candidate summary includes: comparing the key entities of the source text data with the key entities of the candidate summary; replacing the key entities of the candidate summary with the key entities of the source text data, and using the obtained candidate summary as a content fidelity representation to complete the verification of the key entities.

Images