An evidence and assertion alignment method for large language model literature review generation

Abstract

The application discloses a method for aligning evidence and assertion for large language model literature review generation. The ordered event relations and original evidence sentences verified, standardized and label-mapped are extracted from the theme literature, and the evidence relation graph is constructed according to the literature support and the text consensus confidence, and the source mapping is reserved. The candidate review paragraph with actual reference is decomposed into atomic propositions, the event matching and actual reference limitation are performed on the verifiable propositions, the calibration relation state probability is output by a semantic alignment model, the reference evidence edge is selected, and the proposition level evidence alignment score is calculated according to the non-symmetrical rule. The composite score is calculated in combination with the relative fluency, the propositions with evidence missing, relation semantic conflict and strength exaggeration are modified and supplemented, the local evidence relation graph is updated and iteratively generated, and the literature review text is output. The application can reduce the evidence-assertion mismatch in the generated review, and provides a new design idea for improving the reliability of the content of the large language model automatic review generation.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and intelligent text generation, and in particular relates to a method for aligning evidence and assertions in literature review generation for large language models. Background Technology

[0002] In recent years, large language models have been widely used for automatic literature review generation. However, because large language models generate content by predicting subsequent lexical units, their optimization objectives focus on the probabilistic reasonableness and linguistic coherence of the text, and do not require that every relational assertion be implied by actual cited evidence. This often leads to evidence-assertion mismatch between the relational assertions in the generated text and the actual content of the cited literature. This specification defines a proposition that states the functional, temporal, or directional relationship between two events or conceptual entities as a relational assertion, and classifies mismatches into three categories: lack of evidence, semantic conflict of relation, and exaggerated strength. Lack of evidence refers to the generated assertion lacking verifiable citation evidence, or its scope of application exceeding the coverage of the cited evidence; semantic conflict of relation refers to the inconsistency between the cited evidence and the generated assertion in terms of the nature or polarity of the relation; exaggerated strength refers to the generated assertion's expression strength exceeding the support of the original text.

[0003] Existing automated review generation methods primarily improve literature coverage and article organization through retrieval enhancement, structure planning, proxy collaboration, and long context processing. Their evaluation mainly focuses on overall indicators such as literature relevance, citation coverage, and citation usability. These methods do not yet use relational assertions, cited evidence, and relational tags as unified verification units and feedback signals, making it difficult to determine whether the generated statements conform to the relational nature, polarity, and strength supported by the evidence.

[0004] Existing research on illusions and source consistency in large language models distinguishes between internal illusions that contradict the source and external illusions that cannot be verified by the source, and evaluates the correspondence between generated text and cited sources from the perspectives of citation correctness and citation completeness. However, simply judging whether an assertion is supported by the source or whether it conflicts with the source is insufficient to identify exaggerated strength. Some assertions, although supported by citations and not directly conflicting with the evidence, may present weaker associations or contributions as deterministic causality or decisive effects. Summary of the Invention

[0005] This invention proposes a method for aligning evidence and assertions in literature review generation for large language models. It utilizes relation-level evidence verification to organize cross-documentary relational evidence into traceable constraints.

[0006] By using evidence alignment scoring, the nature of relations, polarity of influence, and strength of expression are verified within the actual citation range of candidate propositions. The verification results are then fed back to the candidate selection, targeted correction, and evidence supplementation processes, ensuring that the relational assertions in the generated text are commensurate with the cited evidence.

[0007] This invention provides a method for aligning evidence and assertions in literature review generation for large language models, comprising the following steps:

[0008] 1) Extract verified and standardized ordered event relationships and original evidence sentences from the subject literature, assign relationship labels according to the nature of the relationship, the polarity of the role and the strength of the expression commitment, calculate the text consensus confidence based on cross-document support, construct an evidence relationship diagram with relationship labels, text consensus confidence, source literature and original evidence sentences, and use it for review outline planning and local evidence location;

[0009] 2) Decompose candidate review paragraphs with actual citations into atomic propositions and determine the scope of verification; perform event semantic matching on verifiable propositions, limit candidate evidence edges by the intersection of the actual citation set of the proposition and the source set of relation edges, select benchmark evidence edges based on the calibration relation state probability and text consensus confidence, and calculate proposition-level evidence alignment scores according to asymmetric rules.

[0010] 3) Aggregate the proposition-level evidence alignment score and relative fluency to form a composite score and select the best candidate; based on the evidence alignment results, perform expression downgrading, rewriting or deletion based on the original evidence sentences, and targeted supplementary retrieval on propositions with strength upgrade, relationship conflict and evidence missing, respectively; update the local evidence relationship graph and repeat candidate generation, evidence alignment scoring and text refinement until the output conditions are met or the maximum number of refinement rounds is reached, and then output the final literature review text.

[0011] Furthermore, step 1) includes relation record extraction and verification, event normalization, relation label mapping, cross-document aggregation, and evidence relation graph construction.

[0012] The subject literature collection is denoted as ,in express Single document within the document. Document identification. For explicitly stated ordered head events, relational expressions, and ordered tail events, the corresponding original text evidence sentences are located, and only relational records whose event location, event order, and relational semantics have been verified are retained. Head events and tail events are normalized by using pronouns, abbreviations, full names, and synonyms to obtain standardized head events. and normalized tail events A unique relation label is assigned based on the relational expression, the original evidence sentence, and the normalized event. .

[0013] The strength of the representation of relational label space and function relations is defined as follows:

[0014]

[0015]

[0016]

[0017] Wherein, Strong, Supportive, and Weak represent a deterministic positive effect, a contributing or promoting effect, and a possible effect with modality qualifiers, respectively; Temporal represents a temporal relationship, and Opposite represents a negative effect relationship; the symbols ≻ and ≺ represent the strength of the expression being higher and lower, respectively; Temporal and Opposite do not participate in the order of the strengths.

[0018] Documents The set of relation triples that have been verified, normalized, and have unique relation labels is denoted as . For the same normalized ordered event pair and relation label, the document support is defined as the number of documents counted according to different documents:

[0019]

[0020] in, This is an indicator function; when the same normalized event pair and relation label are repeatedly stated in the same document, only one support is counted.

[0021] For normalized event pairs supported by at least one literature review, the relation label The text consensus confidence level is defined as:

[0022]

[0023] in, For the summation variable that iterates through the relation label space, Indicates the number of relation tags. For smoothing parameters.

[0024] Let the threshold for retaining relation edges be denoted as . , retain satisfaction and The relation record is used as a relation edge; the first The relationship edge is denoted as ,in and They are the first Normalization of head and tail events of a relation edge. and These are the relation label and text consensus confidence of the edge, respectively.

[0025] Further, the evidence relationship diagram described in step 1) is defined as:

[0026]

[0027] in, To standardize the set of event nodes, Let be the set of relation edges. This maps the relationship edges to source documents and original text evidence sentences. The topic-level evidence relationship diagram is used for review outline planning; for the first... Subsection A local evidence relationship diagram is constructed from the literature set of this subsection, which is used for evidence location and proposition scoring.

[0028] Furthermore, step 2) includes proposition verification scope determination, event matching, actual reference limitation, semantic alignment, and asymmetric proposition-level scoring.

[0029] Candidate review paragraphs are broken down into atomic propositions; atomic propositions that can be attributed to specific literature evidence and state specific research conclusions, numerical values, comparisons, relationships between events, or scope of application are identified as verifiable propositions; for atomic propositions that contain both verifiable and general content, the verifiable content is broken down into independent verifiable propositions.

[0030] The first A verifiable proposition is denoted as The relation labels obtained by mapping the relation representation are denoted as... A text encoder is used to map the proposition head, tail events, and local evidence relationship graph nodes to the same semantic space. This is done when the semantic similarity is not lower than the event matching threshold. Event matching is accepted at that time.

[0031] Those that can be attributed to And the actual set of citations that can be located in the current subsection's document set is denoted as Filter the source document set from the matching relationship edges. Relational edges that have intersections are considered candidate evidence edges, and their index set is denoted as... If the event match is not reached... If the actual reference set is empty or the intersection of the source sets is empty, then let .

[0032] when When not empty, the semantic alignment model will With the Joint coding of candidate evidence edges, in Output a temperature-scaled and calibrated probability vector in the three-class label space. Wherein, Aligned indicates supporting evidence, meaning both points to the same event relationship and there is no conflict in the nature or polarity of the relationship; Contradict indicates conflicting evidence, meaning the two are inconsistent in the nature or polarity of the relationship; Neutral indicates insufficient evidence, meaning the candidate evidence edge is insufficient to support or refute the verifiable proposition. The sum of the three components of the probability vector is 1. The benchmark evidence edge is selected according to the following formula:

[0033]

[0034] Let the relation labels of the benchmark evidence edges and the text consensus confidence be denoted as follows: and ;make for The relationship state with the highest probability. empty or At that time, mark the lack of evidence. Otherwise .

[0035] when When not empty, let , , The propositional evidence alignment score is defined as:

[0036]

[0037] in, This is a conservative downgrade discount factor; , and All are greater than 0, representing penalty coefficients for intensity upgrades, rewriting temporal relationships as having a positive effect, and other conflicting properties or effects of relationships. Propositions with consistent relationships and comparable strengths receive positive scores, conservative downgrades receive discounted positive scores, and intensity upgrades, temporal rewriting, and other relationship conflicts each receive different negative scores.

[0038] Furthermore, step 3) includes candidate composite scoring, output determination, targeted correction, supplementary retrieval, and local evidence relationship graph update.

[0039] Candidate review paragraphs in the same generation round are denoted as Let the set of verifiable propositions be denoted as and order The perplexity of candidate paragraphs is calculated using a language model. The relative fluency score is obtained by referencing the average perplexity of candidates in the same round. .

[0040] Candidate review paragraphs The composite score is defined as:

[0041]

[0042] in, The weights for the propositional evidence alignment score and the relative fluency score; The second branch is used, and division by zero does not occur. The optimal candidate for the current generation round is selected based on the composite score; if it contains no verifiable propositions, it is output directly, or if the composite score reaches the score acceptance threshold and all verifiable propositions satisfy the condition... and Output in advance.

[0043] Output conditions not met and maximum refining rounds not reached. When the intensity of an argument is upgraded, its wording is downgraded; when the polarity of a relational property or function is conflicting, the original evidence sentence corresponding to the baseline evidence is rewritten or deleted; for... Proposition 1 executes targeted supplementary retrieval.

[0044] The first Wheel Festival The collection of documents is denoted as The collection of documents obtained from this round of supplementary searches is denoted as And record the subsection document budget as Sort the subsections by relevance to the search and update the document set within budget:

[0045]

[0046] in, This indicates that the search results with the highest relevance should be retained. This document contains [number] documents. For newly added documents, the extraction, verification, normalization, aggregation, and screening processes described in step 1) are repeated to update the local evidence relationship diagram. This achieves [goal / goal]. If the results are not yet output, prioritize outputting the proposition with the highest composite score and all verifiable propositions that are satisfied. and If the candidate does not exist, delete the verifiable propositions that have not passed verification from the highest-scoring candidate and then output them.

[0047] The method of this invention can establish traceable constraints on cross-document relationship evidence and feed the verification results back to the generation process, which has the following advantages compared with the prior art:

[0048] 1. This invention uses ordered event pairs to represent cross-document relationship evidence, establishes a relationship label space according to the nature of the relationship, the polarity of its function, and the intensity of its expression, and characterizes the degree of consistency of cross-document relationships on the same relationship through textual consensus confidence, providing constraints for outline planning and local evidence location;

[0049] 2. This invention uses the actual citation of candidate propositions as the scope of evidence, and combines calibration relation state probability, relation label and text consensus confidence to identify evidence missing, relation semantic conflict and strength exaggeration through asymmetric scoring;

[0050] 3. This invention feeds back the relation-level verification results to candidate selection, targeted correction and evidence supplementation through a generation, scoring and refining cycle, forming a generation closed loop under a clear termination condition. Attached Figure Description

[0051] Figure 1 is a flowchart of the overall process of the method of the present invention.

[0052] Figure 2 shows the overall quality comparison between the method of the present invention and existing methods on the SurveyBench benchmark.

[0053] Figure 3 shows a comparison of the evidence mismatch rate between the method of the present invention and existing methods. Detailed Implementation

[0054] To make the above-mentioned features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0055] The literature review generation method designed in this invention organizes cross-document relationship evidence using an evidence relationship diagram, verifies candidate propositions using an evidence alignment scoring model, and uses the verification results for candidate selection and iterative refinement. The specific method flow is as follows: Figure 1 As shown, its main steps include:

[0056] Step 101: Obtain the literature collection corresponding to the research topic. Extract ordered head events, relational expressions, ordered tail events and their original text evidence sentences; perform event location, event order and relational semantic verification on the extraction results, and only send the relational records that pass all three verifications to step 102;

[0057] Step 102: Normalize the references, abbreviations, full names, and synonyms of the verified head and tail events, and assign unique relation labels according to the relation label space. When mapping, first determine the nature and polarity of the relation, and then determine the strength of the relational expression based on the relational predicate, modal words, scope words, and reserved modifiers.

[0058] Step 103: According to the support of the literature and text consensus confidence The formula performs cross-document aggregation of relation records, and then retains the relation edge based on the threshold. Filter relation edges and save the mapping of each relation edge to the source document and the original evidence sentence;

[0059] Step 104: Construct a topic-level evidence relationship graph using normalized events as nodes and filtered relationship records as edges; convert the relationship edges sorted by text consensus confidence into natural language relationship priors to generate a review outline, and construct local evidence relationship graphs for each subsection in the outline.

[0060] Step 201: Decompose the candidate review paragraphs into atomic propositions and identify verifiable propositions according to the aforementioned judgment criteria; for atomic propositions that contain both verifiable content and general content, separate the verifiable content and verify it separately.

[0061] Step 202: Calculate the cosine similarity between the proposition head and tail events and the nodes in the local evidence relationship graph using a text encoder; only when the maximum similarity at both ends reaches... Timely event matching, then with The intersection of the source document set with the relation edge limits the candidate evidence edge, forming an index set. ;

[0062] Step 203, when When not empty, the semantic alignment model output The calibration probability of each candidate evidence edge being in one of the three relational states of evidence support, evidence conflict, and insufficient evidence. Determined according to the formula for selecting the basis evidence. and obtain accordingly and ;

[0063] Step 204: Align scores according to propositional evidence. The segmented formula is used to complete the scoring, identifying situations where evidence is missing, semantic conflicts arise in relationships, and the strength of the statement exceeds the scope of the evidence. It can verify the semantic alignment between propositions and candidate evidence edges, the selection of benchmark evidence edges, and the proposition-level evidence alignment scoring process.

[0064] Step 301: Within the same generation round, generate a set of candidate review paragraphs for the current subsection. .in, This refers to the number of candidate review paragraphs generated for the current subsection in this generation round. This serves as the index for candidates in the same round. The perplexity of each candidate is calculated using a language model, and the relative fluency score is calculated according to the following formula:

[0065]

[0066] For any candidate in the candidate summary segment set Its relative fluency score is defined as:

[0067]

[0068] in, For the same wheel The average perplexity of each candidate The slope of the perplexity mapping curve is used. The composite scoring formula is used to calculate the slope of each candidate review paragraph. The candidate with the highest composite score is selected as the optimal candidate for this round; when the optimal candidate does not contain any verifiable propositions, it is output directly, or when its composite score reaches the score acceptance threshold and all verifiable propositions have passed the evidence verification, it is output in advance.

[0069] Step 302: Downgrade the expression of the intensity-upgraded proposition; rewrite the proposition with the temporal relationship as a positive effect into a temporal expression or delete the unsupported expression of effect; rewrite or delete other propositions with conflicting relationship properties or polarities of effect based on the original evidence sentence corresponding to the baseline evidence edge;

[0070] Step 303, for The proposition of 1 is used to construct a targeted retrieval formula based on its head event, tail event, relational expression and subsection theme; supplementary documents are merged with the current subsection document set, deduplicated and sorted by relevance, the subsection document set is updated, and the local evidence relationship diagram is reconstructed;

[0071] Step 304: Re-execute candidate review paragraph generation, evidence alignment scoring, and text refinement using the updated local evidence relationship graph; terminate early according to the output conditions described in the invention, achieving... The final candidate will be output according to the aforementioned catch-all rule.

[0072] This invention has been validated on the publicly available SurveyBench benchmark. Figure 2 The comparison results between the present invention and existing automatic review generation methods in terms of reference quality, outline quality, and content quality show that the present invention achieves high overall quality under both model configurations, indicating that the introduction of relation-level evidence constraints does not reduce the overall quality of the generated review. Figure 3 The results of evidence-assertion alignment evaluations using different methods demonstrate that this invention effectively reduces the evidence-assertion mismatch ratio while maintaining the density of similar relational assertions. In the ablation experiment, after sequentially adding the evidence relationship graph, the evidence alignment scoring model, and the generation-scoring-refining loop, the evidence relationship graph primarily improves outline planning and literature coverage; the evidence alignment scoring model further improves candidate content and citation quality; and the generation-scoring-refining loop continues to improve content quality and literature coverage through targeted correction and evidence supplementation.

[0073] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for aligning evidence and assertions in literature review generation for large language models, characterized in that, include: A. Extract verified and standardized ordered event relationships and original evidence sentences from the subject literature, calculate the text consensus confidence based on cross-document support, and construct a directed labeled multi-evidence relationship graph with standardized events as nodes and relationship records carrying relationship labels, text consensus confidence, source literature and original evidence sentences as edges. B. Generate candidate review paragraphs with actual citations and decompose them into atomic propositions; perform two-end event matching on verifiable propositions, limit candidate evidence edges by the intersection of the actual citation set of propositions and the source set of relation edges, select benchmark evidence edges based on the calibration relation state probability and text consensus confidence, and calculate proposition-level evidence alignment scores according to the asymmetric rules of conservative downgrading to obtain a discount positive score, intensity upgrading and time sequence rewriting to obtain different negative penalties, and other relation conflicts to obtain conflict penalties. C. Aggregate the proposition-level evidence alignment score and relative fluency to form a composite score and select the best candidate; for candidates that do not meet the output conditions, perform expression downgrading, rewriting or deleting based on the original text evidence sentences, and targeted supplementary retrieval according to the lack of evidence marker and relationship conflict type, update the local evidence relationship graph within a fixed literature budget and generate iteratively; output when the evidence completeness and scoring conditions are met, and perform fallback selection or delete propositions that have not passed verification before outputting when the maximum refinement round is reached.

2. The method for aligning evidence and assertions in literature review generation for large language models according to claim 1, characterized in that, Step A further includes the following steps: A1. Identify explicitly stated ordered head events, relational statements, ordered tail events, and original text evidence sentences from paragraphs of the subject literature; retain only relational records where event location, event order, and relational semantics have been verified, and normalize the head events and tail events by using abbreviations, full names, and synonyms to obtain standardized head events. and normalized tail events ; A2. Based on the aforementioned relational description, original evidence sentences, and normalized events, assign a unique relational label to each relational record. The order of commitment strength for relational label space and positive action relations is defined as follows: Among them, Strong, Supportive, and Weak represent positive relationships with different expression intensities, Temporal represents temporal relationships, and Opposite represents negative relationships; A3. Let D be the set of subject literature. For the same normalized ordered event pair and relation label, count the number of different documents to obtain the literature support: in, for Single article in the text For literature The set of relational triples is verified and standardized in the Chinese language. This is an indicator function; when the same normalized event pair and relation label are repeatedly stated in the same document, only one support is counted. A4. For normalized event pairs supported by at least one document, calculate the relation label according to the following formula. Text consensus confidence level: in, For the summation variable that iterates through the relation label space, This indicates the number of labels in the relation label space. For smoothing parameters; A5, Retain satisfaction and The relation record is used as the relation edge, where A threshold is reserved for the relation edges; each relation edge carries a normalized head event, relation label, normalized tail event and text consensus confidence, and retains the mapping from the edge to the source document and the corresponding original text evidence sentence, thereby forming the directed labeled multiple evidence relation graph.

3. The method for aligning evidence and assertions in literature review generation for large language models according to claim 1, characterized in that, Step B further includes the following steps: B1. Decompose candidate review paragraphs into atomic propositions; identify atomic propositions that can be attributed to specific literature evidence and state specific research conclusions, numerical values, comparisons, relationships between events, or scope of application as verifiable propositions; for atomic propositions that contain both verifiable and general content, break down the verifiable content into independent verifiable propositions; A verifiable proposition is denoted as And map their relational representations to relational labels. ; B2, will The nodes in the head event, tail event, and local evidence relationship graph are mapped to the same semantic space and event matching is performed; the event matching is accepted when the semantic similarity of both the head event and the tail event reaches the event matching threshold; from the relationship edges of the two-end matching, the source literature set and... The edges whose actual cited reference sets intersect are used as candidate evidence edges, and their index sets are denoted as... and will the The relation labels and text consensus confidence scores of the candidate evidence edges are denoted as follows: and ; If empty, mark as lacking evidence. ; B3. When When not empty, obtain using semantic alignment model With the The calibration probabilities of candidate evidence edges in three relational states—supportive, conflicting, and insufficient—are denoted as Aligned, where supportive means the two edges point to the same event relation and there is no conflict in relational nature or polarity; conflicting means the two edges are inconsistent in relational nature or polarity, denoted as Contradict; and insufficient means the candidate evidence edge is insufficient to support or refute the verifiable proposition, denoted as Neutral. The calibration probability vectors for the three relational states are denoted as follows: The sum of its three probability components is 1; the baseline evidence edge is selected according to the following formula: Let the relation labels of the benchmark evidence edges and the text consensus confidence be denoted as follows: and ;make for The relationship state with the highest probability; season Otherwise ; B4, Order , , The propositional evidence alignment score is calculated using the following formula: in, and These respectively indicate that the strength of the statement is lower than and higher than the baseline evidence edge; The discount factor is a conservative downgrade factor. , and All are greater than 0, representing the penalty coefficients for intensity upgrades, rewriting temporal relationships into positive effects, and other relationship conflicts, respectively.

4. The method for aligning evidence and assertions in literature review generation for large language models according to claim 1, characterized in that, Step C further includes the following steps: C1. Denote the candidate review paragraphs as... Let the set of verifiable propositions be denoted as and order |;The aggregated propositional evidence alignment score, combined with the relative fluency score, is calculated according to the following formula. Composite score : in, The relative fluency score is obtained based on the relative perplexity of candidates in the same round. The weights for the propositional evidence alignment score and the relative fluency score; when When the composite score is obtained, it is weighted by the average evidence alignment score and the relative fluency score of the verifiable proposition; when In this case, the composite score only includes the relative fluency item to avoid division by zero; C2. Select the optimal candidate review paragraph for the current generation round based on the composite score; output it directly if it does not contain verifiable propositions, or output it when its composite score reaches the score acceptance threshold and all verifiable propositions are satisfied. and Output in advance; C3. When the optimal candidate summary paragraph does not meet the output conditions and the maximum refinement round has not been reached, propositions with statement strength exceeding the range of evidence support are downgraded; propositions with relational properties or polarity inconsistent with the evidence are rewritten or deleted based on the original evidence sentence corresponding to the baseline evidence edge. The proposition is used to perform targeted supplementary retrieval; the supplementary retrieval documents are merged with the current subsection documents, sorted by retrieval relevance, and the subsection document set is updated within a fixed document budget to reconstruct the local evidence relationship graph; C4. Regenerate candidate review paragraphs using the updated local evidence relationship diagram and perform evidence alignment scoring and text refinement; if the maximum refinement rounds are not completed before output, prioritize outputting the paragraph with the highest composite score and all verifiable propositions satisfied. and If no candidate exists, select the candidate with the highest composite score, delete the verifiable propositions that have not passed verification, and then output the result.

Images