An evidence chain-based verifiable large model retrieval enhancement generation system and method

Abstract

The application relates to the technical field of natural language processing, in particular to an evidence chain-based verifiable large model retrieval enhancement generation system and method, which comprises the following steps: receiving an initial query, identifying the ambiguity and information gap of the initial query in combination with associated retrieval documents, and generating a supplementary query set; generating a cited candidate answer based on the initial query, the supplementary query and the corresponding retrieval document, and verifying the information supportability of the cited candidate answer; extracting supportable information from the verified candidate answer and constructing a hierarchical attribution mapping relationship; integrating the above information to form to-be-evaluated information; if the to-be-evaluated information currently verified satisfies a preset sufficiency condition, integrating a generated preliminary answer and the to-be-evaluated information to synthesize a target answer. The application effectively solves the defects of the traditional RAG system, such as information integration fragmentation, one-sided query answering, inefficient attribution and excessive dependence on retrieval content, and has the advantages of comprehensive answer, verifiability and lightweight deployment.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a verifiable large model retrieval enhancement generation system and method based on evidence chains. Background Technology

[0002] Large Language Models (LLMs), such as DeepSeek, ChatGPT, and Gemini, have demonstrated broad application potential across multiple domains due to their powerful language understanding and generation capabilities. However, these models have inherent limitations at the knowledge level: firstly, their knowledge originates from training data, which is static and cannot cover the latest information after the training deadline; secondly, due to the breadth and generality of the training corpus, the models often lack sufficient depth and accuracy in specific professional domains. These limitations are particularly pronounced in high-risk or highly specialized application scenarios.

[0003] To address these shortcomings, the Retrieval-Augmented Generation (RAG) architecture has been proposed and is gradually becoming a key solution. The core of RAG lies in combining parametric models with non-parametric external knowledge sources, enabling the generation process to dynamically incorporate real-time, proprietary, or domain-specific information. In this way, RAG transforms the response mode of large language models from a "closed-book exam" relying on internal memory to an "open-book exam" that can actively query external knowledge, significantly improving the accuracy and timeliness of the generated content.

[0004] A typical RAG workflow includes four key steps: First, data ingestion and indexing: Information is acquired from diverse external data sources (such as APIs, databases, and document repositories), transformed into vector representations through an embedding model, and stored in a vector database to build a knowledge index for efficient retrieval. Second, information retrieval: When a user submits a query, the system encodes the query text into a vector and performs similarity matching in the vector database to obtain the information fragments most relevant to the query semantics. Third, suggestion enhancement: The retrieved contextual information is combined with the original query to construct enhanced suggestions with rich background information. This process typically utilizes suggestion engineering methods to ensure the large language model receives sufficient relevant context. Fourth, content generation: The enhanced suggestions are input into the large language model, which combines external retrieval information with its own internal knowledge to generate more accurate, reliable, and context-relevant answers.

[0005] While RAG has made significant progress in enhancing the timeliness and traceability of generated content, existing implementations still face several key challenges:

[0006] Firstly, information integration is fragmented: online information is usually scattered across multiple heterogeneous sources, while existing RAG systems rely heavily on similarity retrieval and tend to focus on a few top-ranking documents, making it difficult to effectively integrate fragmented knowledge from multiple sources. Users still need to piece together complete answers themselves.

[0007] Secondly, the one-sidedness of responses to vague questions: When users ask relatively vague or open-ended questions, the existing "retrieval-generation" mechanism is prone to being limited to one aspect of the question, resulting in a narrow perspective in the answers and making it difficult to fully cover the multiple dimensions of the question.

[0008] Third, the inefficiency of the citation attribution mechanism: Although the source of the citation increases transparency, users still need to check the original text to verify the content, which is a heavy cognitive burden. The directness and efficiency of citation still need to be improved.

[0009] Fourth, over-reliance on retrieved content: The current RAG mechanism heavily depends on the retrieved context during the generation process, which may inhibit the knowledge reserves and reasoning abilities of the large language model itself. When the retrieved content is of low quality or does not contain the correct answer, it may lead to a decline in the quality of the generated content, failing to fully realize the model's potential. Summary of the Invention

[0010] Therefore, this invention aims to solve the technical problems of existing retrieval augmentation generation (RAG) systems when processing user queries, such as fragmented information integration, one-sided answers to ambiguous questions, inefficient citation attribution, and over-reliance on retrieval content that inhibits the reasoning ability of large language models themselves. In this way, it provides a verifiable large model retrieval augmentation generation system and method based on evidence chains.

[0011] Specifically, the evidence chain-based verifiable large model retrieval enhancement generation system includes: a heuristic question generator module, a credible answer verifier module, an attributable answer mapping constructor module, and an adaptive termination evaluator module.

[0012] The heuristic question generator module is used to receive an initial query question, combine it with the associated search documents, identify the ambiguous position and information gap of the initial query question, and generate a set of supplementary query questions.

[0013] The trusted answer verifier module is used to generate candidate answers with citation information based on the initial query question, the set of supplementary query questions and the corresponding search documents, and to verify the information support of the candidate answers;

[0014] The attributable answer mapping constructor module is used to extract supporting information and construct a hierarchical attribution mapping relationship for verified candidate answers.

[0015] The adaptive termination evaluator module is used to integrate the verified candidate answers, their corresponding extracted supporting information, and hierarchical attribution mapping relationships into information to be evaluated, and to determine whether the currently verified information to be evaluated meets the preset sufficiency conditions:

[0016] If the conditions are met, the preliminary answer generated by the heuristic question generator module and the information to be evaluated are integrated to synthesize the target answer;

[0017] If the conditions are not met, the heuristic question generator module is driven to continue iterating and generating new supplementary query questions based on the current information gap.

[0018] In one embodiment of the present invention, the heuristic question generator module receives an initial query question, combines it with associated search documents, identifies ambiguities and information gaps in the initial query, and generates a set of supplementary query questions, including:

[0019] The initial query question is taken as input, and a preliminary answer associated with the initial query question is generated through model parameterized knowledge reasoning of a preset language model.

[0020] Extract the keywords of the initial query question and use them as search terms for the search engine to obtain a set of search documents associated with the initial query question;

[0021] Define the semantic coverage of the initial query question, perform sentence-by-sentence semantic matching and incremental information identification on each document in the search document set, and filter out discrete fragmented information in the search documents that are not in the semantic coverage but are associated with the initial query question or the preliminary answer to obtain an incremental fragmented information set;

[0022] Based on the preliminary answer and the incremental fragmented information set, the information gaps in the initial query question are identified by semantic association mapping between the fragmented information and the initial query question, and candidate supplementary query questions are generated in a targeted manner based on the information gaps.

[0023] All candidate supplementary query questions are semantically redundantly eliminated and information coverage is verified to form a set of structured supplementary query questions that meet the information gap filling requirements.

[0024] In one embodiment of the present invention, the trusted answer verifier module includes: a candidate answer generation unit, a single-sentence verification unit, and a quality assessment unit; wherein,

[0025] The candidate answer generation unit is used to generate an initial answer containing multiple semantically independent sentences based on a single supplementary query question and its corresponding retrieval document in the supplementary query question set, and to associate each sentence with a unique identifier of the retrieval document from which its information source is located;

[0026] The single-sentence verification unit is used to perform support verification on each sentence in the initial answer, in conjunction with the source document it references, to determine whether the content of each sentence can be supported by the referenced source document, and to obtain the verification result of each sentence.

[0027] The quality assessment unit is used to count the number of sentences with a support judgment result of "pass" based on the verification results of all sentences, calculate the overall attribution accuracy of the initial answer, and retain the initial answers with an overall attribution accuracy greater than or equal to a preset quality assessment threshold as candidate answers that meet the quality requirements.

[0028] In one embodiment of the present invention, the single-sentence verification unit performs support verification on each sentence in the initial answer, in conjunction with the source document it references, determining sentence by sentence whether the content of the sentence can be supported by the referenced source document, and obtaining the verification result of each sentence, including:

[0029] Extract each statement to be verified from the initial answer, and based on its preset reference association with the search document, match the unique search source document corresponding to each statement to be verified to construct an input pair of statement to be verified and associated document;

[0030] Each input pair is fed into the preset verification model one by one. The model performs deep matching of the semantics of the statement and the content of the document, and analyzes the supporting logic of the information to obtain the judgment result of whether the statement to be verified is supported by the associated document. At the same time, the judgment reason is generated, and the set of supporting sentences that can support the judgment result is extracted from the associated document.

[0031] The triplet data consisting of the judgment result, the judgment reason, and the supporting sentence set is used as the verification result for each sentence.

[0032] In one embodiment of the present invention, the attributable answer mapping constructor module includes: a support information extraction unit, an evidence integration unit, and a mapping construction unit; wherein,

[0033] The supporting information extraction unit is used to call a preset supporting information extraction algorithm, based on the semantic features of the verified candidate answers and the content relevance of the retrieval source document, to extract a minimum set of supporting sentences that can support the validity of the candidate answers from the original retrieval source document corresponding to the verified candidate answers, as supporting information;

[0034] The evidence integration unit is used to aggregate the supporting information to form a structured evidence summary with semantic consistency and coherent expression.

[0035] The mapping construction unit is used to establish a hierarchical association mapping structure between the structured evidence summary and each supporting information, and between each supporting information and the corresponding original search document.

[0036] In one embodiment of the present invention, the hierarchical association mapping structure includes a first mapping layer and a second mapping layer. The first mapping layer is the mapping relationship between the structured evidence summary and the supporting information, and the second mapping layer is the mapping relationship between a single sentence in the supporting information and the unique identifier of the corresponding original retrieval source document.

[0037] In one embodiment of the present invention, the adaptive termination evaluator module determines whether the currently verified information to be evaluated satisfies a preset sufficiency condition, including:

[0038] Based on at least two of the following parameters—the number of query questions corresponding to additional retrieval needs, the similarity between newly generated query questions and existing query questions, and the model confidence—a dynamic weight fusion algorithm is used to calculate the sufficiency score of the information to be evaluated.

[0039] The sufficiency score is compared with a preset information sufficiency score threshold to generate a judgment result for the information to be evaluated.

[0040] In one embodiment of the present invention, the sufficiency score is calculated using the following formula:

[0041] ,

[0042] Where I is the information sufficiency marker, α is the prediction confidence level corresponding to the information sufficiency label I, where α is the weighting coefficient. This is a quantitative value for the information gap calculated based on the number of additional search questions. , where n is the number of questions that need to be additionally retrieved, as given by the large language model. denoted by , where is the maximum limit on the number of problems; and S is the similarity between the newly generated problem and the existing problem.

[0043] Based on the same inventive concept, this invention also provides a verifiable large model retrieval enhancement generation method based on evidence chains, comprising the following steps:

[0044] S1: Receive the initial query question, combine it with the associated search documents, identify the ambiguous position and information gap of the initial query question, and generate a set of supplementary query questions;

[0045] S2: Based on the initial query question, the set of supplementary query questions, and the corresponding search documents, generate candidate answers with citation information, and verify the information support of the candidate answers;

[0046] S3: For the verified candidate answers, extract supporting information and construct a hierarchical attribution mapping relationship;

[0047] S4: Integrate the verified candidate answers, their corresponding extracted supporting information, and hierarchical attribution mapping relationships into information to be evaluated, and determine whether the currently verified information to be evaluated meets the preset sufficiency conditions:

[0048] If the conditions are met, the generated preliminary answer and the information to be evaluated are integrated to synthesize the target answer;

[0049] If the conditions are not met, return to step S1 and continue iterating to generate a new supplementary query question based on the current information gap.

[0050] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the aforementioned method for enhancing the retrieval of verifiable large models based on evidence chains.

[0051] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0052] This invention proposes an improved framework to address the shortcomings of existing Retrieval Enhancement Generation (RAG) systems, offering several significant advantages: First, by actively identifying the ambiguity and information gaps in the initial query through a heuristic question generator module, and generating supplementary queries by combining fragmented information from the retrieved documents, it effectively solves the problems of fragmented information integration and one-sided answers to ambiguous questions in traditional RAG systems, ensuring the comprehensiveness of information input; Second, relying on the sentence-by-sentence support verification and overall attribution accuracy screening mechanism of the trusted answer validator module, and the "evidence summary → supporting information" constructed by the attributable answer mapping constructor module, it achieves this. The hierarchical attribution mapping relationship of the original search documents avoids blindly following low-quality search content and solves the problem of inefficiency in traditional RAG citation attribution, achieving accurate traceability and high credibility of the answer; thirdly, the adaptive termination evaluator module dynamically judges the iteration termination based on the sufficiency score and preset threshold, ensuring the quality of the answer while avoiding redundant iteration, balancing the answer effect and computational cost, and can be implemented based on small and medium-sized models with low hardware deployment threshold. At the same time, it supports combining with real-time network search to improve the timeliness of the answer, significantly improving the comprehensiveness, reliability and practical value of the RAG system. Attached Figure Description

[0053] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0054] Figure 1 This is a schematic diagram of a verifiable large model retrieval enhancement generation system based on evidence chain provided in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of a verifiable large model retrieval enhancement generation method based on evidence chain provided in an embodiment of the present invention. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0057] Example 1:

[0058] Reference Figure 1 As shown, the present invention provides a verifiable large model retrieval enhancement generation system based on evidence chain, comprising: a heuristic question generator module (HQG), a credible answer verifier module (EAV), an attributable answer mapping constructor module (AMC), and an adaptive termination evaluator module (ATE).

[0059] The heuristic question generator module (HQG) is used to receive an initial query question, combine it with the associated search documents, identify the fuzzy position and information gap of the initial query question, and generate a set of supplementary query questions.

[0060] The Trusted Answer Verifier (EAV) module is used to generate candidate answers with citation information based on the initial query question, the set of supplementary query questions, and the corresponding search documents, and to verify the information support of the candidate answers.

[0061] The Attributable Answer Mapper (AMC) module is used to extract supporting information and construct hierarchical attribution mapping relationships for verified candidate answers.

[0062] The Adaptive Termination Evaluator (ATE) module is used to integrate the validated candidate answers, their corresponding extracted supporting information, and hierarchical attribution mapping relationships into information to be evaluated, and to determine whether the currently validated information to be evaluated meets the preset sufficiency conditions:

[0063] If the conditions are met, the preliminary answer generated by the heuristic question generator module (HQG) and the information to be evaluated are integrated to synthesize the target answer;

[0064] If the conditions are not met, the heuristic question generator module (HQG) is driven to continue iterating and generating new supplementary query questions based on the current information gap.

[0065] As can be seen from the above technical solution, the verifiable retrieval enhancement generation system provided in this embodiment consists of four core modules: Heuristic Question Generator (HQG), Trusted Answer Verifier (EAV), Attributable Answer Map Constructor (AMC), and Adaptive Termination Evaluator (ATE). Each module realizes information flow through data interface and collaboratively completes the closed-loop process of "initial query processing → supplementary query generation → candidate answer verification → attribution map construction → iterative termination judgment → final answer synthesis".

[0066] Furthermore, the heuristic question generator module (HQG) receives the initial query question, combines it with the associated search documents, identifies the ambiguities and information gaps in the initial query, and generates a supplementary query question set through the following specific steps:

[0067] Initial query problem of receiving user input In this embodiment, the user queries " How many the parks are there in place A? Taking "" as an example, we perform standardization processes such as removing syntactic redundancy and unifying capitalization to obtain the standardized query problem "". How many the parks are there in place A? ”;

[0068] The standardized initial query question is used as input, and a pre-defined language model (Llama3-8B model) is invoked. Based on the model's parameterized knowledge reasoning, a preliminary answer is generated. " There are 38 parks in place A ”;

[0069] Extract the keywords from the initial query question ("parks", " place A The search engine constructs search terms, calls the search engine interface to perform a search, and obtains results related to the initial question. Related document collection The documents are deduplicated, formatted, and converted to plain text format.

[0070] The semantic vector of the initial query question is calculated using the Sentence-BERT all-MiniLM-L6-v2 word vector model to define the semantic coverage of the initial query question. Based on the above example, its semantic coverage is determined to be "quantity statistics of region A".

[0071] For the document collection Each document in Sentences are split into sentences using periods, question marks, and exclamation marks as delimiters to obtain sentence sequences. and for each sentence Mark the document ID and location index;

[0072] Iterate through all sentences and, through semantic vector similarity calculation, filter out sentences that are not within the semantic coverage of the initial query but are related to the initial query question. Or preliminary answer The associated discrete fragmented information forms an incremental fragmented information set. : , , This represents the total number of sentences after filtering. The total number of sentences in all documents;

[0073] In this embodiment, the filtered... Contains the sentence " The place A's first six state parks were created in June 1936 "", The place A Department of Conservation and Recreation oversees the state park system "wait;

[0074] Based on the incremental fragmented information set Generate the initial problem The relevant prompt is: "Based on the following information, generate relevant questions to supplement the query 'Number of parks in Region A': 1. The first six state parks in Region A were established in June 1936; 2. The Department of Environmental Protection and Recreation in Region A is responsible for managing the state park system...";

[0075] Input the prompt words into the preset language model (Llama3-8B model) to generate candidate supplementary query questions, including " When were the first state parks in the place A established "?" Which department oversees place A's state park system The process involves calculating the Jaccard similarity between candidate supplementary query questions to eliminate semantically redundant queries; simultaneously, verifying the information coverage of each query to ensure it fills the information gaps in the initial query, ultimately forming a structured set of supplementary query questions. .

[0076] Furthermore, the Trusted Answer Verifier (EAV) module is used to generate candidate answers with citation information and verify their support, including: a candidate answer generation unit, a single-sentence verification unit, and a quality assessment unit. The specific implementation steps are as follows:

[0077] The candidate answer generation unit is configured to: for the supplementary query question set Each supplementary query question Call Elasticsearch to retrieve the corresponding related documents, such as supplementary query questions. Related documents , Additional query questions Related documents , ;

[0078] The fine-tuned Llama3-8B model, optimized through attribution training, is invoked to generate an initial answer containing multiple semantically independent sentences, using a single supplementary query question and its corresponding associated document as input pairs. Furthermore, each sentence is associated with a unique identifier of the retrieved document from which its information source is located; in this embodiment, for... The generated initial answer Sentences " The first six state parks in place A were created in June 1936 The associated document is identified as follows: ;sentence " The establishment of these parks was part of the New Deal program The associated document is identified as follows: ;

[0079] The single-sentence verification unit is configured to: verify the initial answer. Each sentence to be verified Based on its associated document identifier, it matches the corresponding original search document. Construct input pairs of the sentence to be validated and the associated document. ,like ;

[0080] A dedicated validation model (denoted as V), fine-tuned for the "sentence-document support determination" task, will validate each input pair. The model inputs sentences one by one, and through deep matching of sentence semantics and document content and logical analysis of information support, it outputs a triplet of verification results. ,in The result of the binary classification is either "support" or "do not support". For the reasons for the judgment, such as of For "document" The 8th sentence clearly records ' Place A's first six state parks were established in June 1936 ', which is completely consistent with the semantics of the sentence to be verified'; The core supporting sentence set extracted from the associated documents, and ; triple data This serves as the validation result for each sentence.

[0081] The quality assessment unit is configured to: assess all sentences Verification results Statistical support judgment results The number of sentences that "support" the statement, according to the formula Calculate the overall attribution accuracy of the initial answer, where n is the total number of sentences in the draft answer. For indicator functions, when When the time is right, the value is 1;

[0082] Preset quality assessment threshold , calculate and Comparison: If If the answer is deemed of low quality, it means that a good answer cannot be provided based on the existing search results, and the initial answer is discarded; if If the answer is deemed generally credible, the initial answer will be retained as a candidate answer that meets the quality requirements.

[0083] Regarding the verification results Sentences marked "not supported" but whose overall initial answer quality meets the standard are selected based on the reasoning given. The context of the answer and related documents Rewrite it. For example, a certain sentence. " All Place A parks are free to enter The verification result is "not supported," and the reason for this determination is as follows: The original sentence, "The document only states that some state parks are free, while others require an entrance fee," has been rewritten as "...". Many place A's parks are free to enter, while some charge a small entrance fee ".

[0084] To optimize and utilize the model's historical validation answers, a dynamic answer pool is constructed. Candidate answers validated by EAV, along with their corresponding queries, associated document identifiers, and core supporting sentence sets, are stored in the pool to form structured question-answer pairs. This serves as an online knowledge base for subsequent answer synthesis. In this embodiment, the question-answer pairs stored in the answer pool include... wait.

[0085] Meanwhile, in web retrieval scenarios such as fuzzy queries and complex queries, key information supporting the answer is often discretely distributed across multiple heterogeneous documents. Existing citation mechanisms lack a standardized organizational paradigm that balances effective integration of fragmented information with rapid user verification. To address this, this invention proposes a Hierarchical Evidence Consolidation citation processing strategy. This strategy is implemented through an Attributable Answer Mapper (AMC) module. Its core functions include supporting information extraction, hierarchical attribution map construction, and dynamic answer pool maintenance. Specifically, it is collaboratively completed by a supporting information extraction unit, an evidence consolidation unit, and a map construction unit. The implementation steps of each unit are as follows:

[0086] The information extraction support unit is configured to: target the original document Only a small number of key fragments (such as single sentences and phrases) directly support the features of the target answer. A pre-defined supporting information extraction algorithm is invoked. By modeling the correlation between the semantic features of candidate answers and the content of the retrieved source document, a minimal set of core supporting sentences that supports the validity of the candidate answers is extracted and defined as supporting information. ={" The Place A park system currently includes 38 parks " As of 2024, there are 38 state parks managed by the Place A Department of Conservation and Recreation};

[0087] The evidence integration unit is configured to: invoke an evidence aggregation model optimized based on Llama3-8B. For the core support sentence set Semantic association analysis, redundant information removal, and logical coherence processing are performed to integrate the evidence into a structured evidence summary with semantic consistency and expressive coherence. : In this embodiment, the integrated structured evidence summary for:" As of 2024, the Place A park system, managed by the Place A Department of Conservation and Recreation, includes 38 state parks ”;

[0088] The mapping construction unit executes synchronously with the evidence integration unit and is configured to: invoke the attribution mapping construction algorithm to construct a hierarchical association mapping structure of "structured evidence summary → supporting information → original retrieved document". This achieves a balance between information integration and traceability. The hierarchical structure specifically includes two mapping relationships: first, the precise association between structured evidence summaries and supporting information, i.e., <structured evidence summary P, core supporting sentence set E>; second, the mapping between a single sentence in the supporting information and the unique identifier of the original retrieved document, i.e., <a single sentence in the core supporting sentence set, corresponding to the unique identifier of the original retrieved document>; the final output of the complete traceable attribution mapping relationship expression is: ,in This is a set of associations between the core supporting sentences extracted from multiple source documents and their corresponding original documents. This ensures both the readability of the aggregated evidence and preserves the original information source, achieving a dual optimization of information density and fact traceability.

[0089] In our iterative framework, the model determines whether to iterate and retrieve new information multiple times based on whether the currently validated answer and cited content can construct a complete response. However, relying solely on the model's judgment is sometimes unreliable. To address this issue, we introduce a more robust, quantized score-based Dynamic Early-Stopping Strategy to control the iteration process. In the Prompt, we prompt the model to analyze whether the current conditions are sufficient to construct a complete response and allow the model to suggest additional content to retrieve when conditions are insufficient. Since we have a limit on the number of questions that the LLM can generate, we can measure the degree of missing information based on the number of questions provided by the LLM:

[0090] Furthermore, the core design goal of the Adaptive Termination Evaluator (ATE) module is to address the unreliability problem of purely relying on the subjective judgment of the Large Language Model (LLM) in iterative frameworks. By introducing a dynamic early termination strategy based on quantization scores, it achieves precise control over the iteration process. The specific logic for evaluating whether the currently verified information to be evaluated meets the preset sufficiency conditions is as follows:

[0091] Specifically, the ATE module first performs the core parameter acquisition operation: firstly, it prompts the Llama3-8B model to analyze the current information gap and generate the number of additional questions n that need to be retrieved; secondly, it calculates the set of supplementary query questions. The Jaccard similarity S with existing queries (including the initial query and historical supplementary queries) is used to characterize the semantic differences between iterative exploration of new directions and the original problem; thirdly, the information sufficiency label I of LLM for the current information sufficiency and the corresponding prediction confidence are obtained. The information sufficiency marker I is used to mark the information status, and the confidence level is used to quantify the reliability of the judgment result.

[0092] Information sufficiency is labeled as a binary variable (with a value range of 1 / 2). This is used to characterize the degree to which the current set of retrieved information satisfies the initial query, and its determination is achieved through standardized suggestion engineering and LLM inference:

[0093] Triggering conditions are determined by the system automatically constructing a "initial query - retrieval information pair" of prompts, which includes task instructions, initial query text, and the current set of retrieval fragments (including source annotations and key information extracts).

[0094] Decision model: The Llama3-8B model, which is of the same origin as the Heuristic Question Generator (HQG) module, ensures the consistency between the decision logic and the query generation logic;

[0095] Judgment process: After receiving the prompt words, the Llama3-8B model outputs a unique judgment label ("sufficient information" or "insufficient information") through semantic understanding, demand matching and completeness reasoning.

[0096] Label assignment rule: If the model outputs "sufficient information", then I=1; if the output is "insufficient information", then I=0.

[0097] Example suggestion keyword construction guidelines: For the initial query " How many parks are there in place A? If the current search segment already contains "the park system in region A in 2024 includes 38 parks" and official verification information, the prompt word is defined as: "Task: Determine whether the following search information is sufficient to completely answer the query question. Query question [] How many Are parks located in Place A? The current search segments are: 1. 'As of 2024, the place A parksystem includes 38 parks' (Source: place A Department of Conservation and Recreation); 2. 'The place A Department of Conservation and Recreation confirms the 38-park statistic' (Source: 2024 A Region Natural Resources Bulletin). Requirements: If the information covers the core query requirements and is not missing any key information, output the label 'Information Sufficient'; if there are gaps in core information or data uncertainty, output the label 'Information Insufficient'.

[0098] The prediction confidence score Conf (range: [0,1]) is the probability confidence score of the corresponding judgment label output by the Llama3-8B model when performing the above information sufficiency binary classification. It is used to quantify the degree of confidence the model has in its own judgment results.

[0099] When I=1 This represents the prediction confidence level at which the model is certain that "the information is sufficient"; when I=0, This indicates the confidence level of the prediction, suggesting that the model is certain that "the information is insufficient."

[0100] The acquisition path is as follows: Based on the probability output interface of the Llama3-8B model, while the model generates the decision label, the corresponding class probability value output by its softmax layer is extracted and directly used as... The quantization results require no additional training or post-processing.

[0101] Based on the parameters obtained above, ATE calculates the sufficiency score of the information to be evaluated using a dynamic weight fusion algorithm. :

[0102] ,

[0103] Where α is a preset weighting coefficient. This is a quantitative value for the information gap calculated based on the number of additional search questions. The calculation method is as follows: , where n is the number of questions that need to be additionally retrieved, as given by the large language model. The maximum number of questions is preset; S is the Jaccard similarity between newly generated query questions and existing query questions; this formula is dynamically switched using the label I: when the model determines that the answer is sufficient (I=1), the sufficiency score is mainly determined by the model confidence. Dominant; when the model determines insufficient information (I=0), the score depends on the potential value of continuing iterations to fill the information gap and explore novel content. and Collaborative quantification.

[0104] The calculated sufficiency score Compared with the preset information sufficiency score threshold Perform numerical comparisons:

[0105] like This indicates that there is still a gap in the current information. The ATE module will drive the Heuristic Question Generator (HQG) module to continue generating new supplementary queries based on the current information gap, repeating the aforementioned retrieval-verification-mapping process.

[0106] like If the current information to be evaluated meets the sufficiency requirement, the preliminary answer generated by the heuristic question generator module is integrated with the information to be evaluated to form a complete and traceable target answer.

[0107] To verify the comprehensive performance of the verifiable large model retrieval enhancement generation framework based on evidence chains proposed in this invention, this embodiment selects three mainstream question answering evaluation datasets: the summary question answering dataset ASQA, the popular explanation question answering dataset ELI5, and the expert-level professional question answering dataset EXPERTQA. The framework is compared and evaluated with existing state-of-the-art methods in the field, including both prompt-based and training-based methods. Performance is measured from the two core dimensions of answer correctness and citation quality.

[0108] The prompt-based approach employs the standard RAG settings under the Automated Large Language Model Citation Evaluation Framework (ALCE) benchmark, using Llama2-7B, Llama3-8B, and GPT-4.1 as base models. The training-based approach includes Self-Reflective Retrieval Enhancement Generation (Self-RAG) and Rejection Sampling and Reinforcement Learning (RS+RL) methods fine-tuned on Llama2-7B. Furthermore, the approach using fine-grained factual evidence-based citation generation (FRONT) is validated on both Llama2-7B and Llama3-8B, and the performance ceiling under the Oracle (perfect retrieval) setting based on Llama3-8B is reported.

[0109] The correctness of the model's responses is evaluated using different metrics for different tasks: the ASQA dataset uses exact match recall (EM Rec), the ELI5 dataset uses claim correctness recall (Claim Rec), and the EXPERTQA dataset uses text similarity (RougLsum). The citation quality of the model is uniformly measured using classic metrics in the retrieval field, including recall (Rec), precision (Pre), and their harmonic mean F1 score.

[0110] The experiment strictly followed the standard evaluation process in the field of retrieval enhancement generation technology to ensure the objectivity and cross-method comparability of the results. The specific experimental results are shown in Table 1.

[0111] Table 1

[0112]

[0113] In evaluating the processing capabilities for fuzzy queries, the ASQA dataset presents a core challenge due to the openness and multi-dimensionality of its query expressions. Our framework, relying on a unique iterative question-answering driven mechanism, achieves an outstanding score of 51.5 on the answer correctness metric (EM Rec). This result not only significantly surpasses comparable models with similar parameter counts but also outperforms state-of-the-art methods based on GPT-4.1, fully demonstrating the architectural innovation and technological superiority of our framework in deep semantic understanding and cross-source fragmented information integration.

[0114] In the evaluation of answer accuracy metrics (Claim Rec, RougeLsum), the ALCE method corresponding to the GPT-4.1 model with its ultra-large number of parameters showed a slight advantage on the ELI5 and EXPERTQA datasets. This phenomenon is consistent with the laws of technological cognition and is attributed to the native knowledge reserves and strong reasoning capabilities of the GPT series models supported by massive training data. It is worth noting that this framework is implemented based on the lightweight Llama3-8B model, which reduces hardware resource consumption by more than 60% compared to the GPT-4.1 model, demonstrating a significant advantage in lightweight deployment. More importantly, even with a significant difference in the scale of model parameters, this framework can still achieve performance superiority in core scenarios such as fuzzy query processing (e.g., the ASQA dataset), further highlighting the advanced nature and technological innovation of the invention's architecture design.

[0115] The core technical advantages of this framework lie in the accuracy of answer attribution and the reliability of citations. Experimental results show that in the Citation F1 score evaluation of three datasets, this framework achieves an absolute leading performance at the state-of-the-art (SOTA) level in the current field: on the ASQA dataset, the framework achieves a Citation F1 score of 82.6, a significant improvement of 14.2 points compared to the second-best comparative method's 68.4; on the ELI5 and EXPERTQA datasets, the framework achieves Citation F1 scores of 63.4 and 63.0 respectively, nearly doubling the performance of existing methods. These results fully validate the effectiveness of the hierarchical attribution mapping technology and embedded real-time verification mechanism adopted by this framework, ensuring that the generated answers have extremely high verifiability and accurate traceability.

[0116] Furthermore, this embodiment also conducts comparative tests on two typical knowledge source access modes (static knowledge base access mode and real-time web search access mode). The static knowledge base used is the 2018 version of Wikipedia, and the real-time web search is implemented using a standard search engine interface. Experimental data shows that after enabling the real-time web search mode, this framework achieves significant improvements in both the information timeliness index and the citation accuracy index of the entire dataset in the long text question answering scenario (ELI5 dataset), verifying the framework's efficient integration capability of dynamic and real-time external information. In the fuzzy query processing scenario based on a specific domain static knowledge base (ASQA dataset), the answer correctness index of the static knowledge base access mode is better. It is speculated that this is because the static knowledge base has lower information noise and stronger domain relevance, resulting in a higher knowledge matching degree with the evaluation dataset.

[0117] Example 2:

[0118] Based on the same inventive concept as the system described in Embodiment 1, this invention also provides a verifiable large model retrieval enhancement generation method based on evidence chains, such as... Figure 2 As shown, it includes the following steps:

[0119] S1: Receive the initial query question, combine it with the associated search documents, identify the ambiguous position and information gap of the initial query question, and generate a set of supplementary query questions;

[0120] S2: Based on the initial query question, the set of supplementary query questions, and the corresponding search documents, generate candidate answers with citation information, and verify the information support of the candidate answers;

[0121] S3: For the verified candidate answers, extract supporting information and construct a hierarchical attribution mapping relationship;

[0122] S4: Integrate the verified candidate answers, their corresponding extracted supporting information, and hierarchical attribution mapping relationships into information to be evaluated, and determine whether the currently verified information to be evaluated meets the preset sufficiency conditions:

[0123] If the conditions are met, the generated preliminary answer and the information to be evaluated are integrated to synthesize the target answer;

[0124] If the conditions are not met, return to step S1 and continue iterating to generate a new supplementary query question based on the current information gap.

[0125] Example 3:

[0126] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program described in Embodiment 2, it implements the aforementioned method for enhancing the retrieval of verifiable large models based on evidence chains.

[0127] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A verifiable large model retrieval enhancement generation system based on evidence chains, characterized in that, include: The heuristic question generator module is used to receive an initial query question, combine it with the associated search documents, identify the ambiguous positions and information gaps in the initial query question, and generate a set of supplementary query questions. The credible answer verifier module is used to generate candidate answers with citation information based on the initial query question, the set of supplementary query questions and the corresponding search documents, and to verify the information support of the candidate answers; The Attributable Answer Map Builder module is used to extract supporting information and construct a hierarchical attribution mapping structure for verified candidate answers. And an adaptive termination evaluator module, used to integrate the verified candidate answers, their corresponding extracted supporting information, and hierarchical attribution mapping structure into information to be evaluated, and to determine whether the currently verified information to be evaluated meets the preset sufficiency conditions: If the conditions are met, the preliminary answer generated by the heuristic question generator module and the information to be evaluated are integrated to synthesize the target answer; If the conditions are not met, the heuristic question generator module is driven to continue iterating and generating new supplementary query questions based on the current information gap; The attributable answer mapping constructor module includes: The information extraction unit is used to call a preset information extraction algorithm to extract a set of minimal supporting sentences that can support the validity of the candidate answer from the original retrieval source document corresponding to the verified candidate answer, based on the semantic features of the candidate answer and the content relevance of the candidate answer. An evidence integration unit is used to aggregate the supporting information to form a structured evidence summary with semantic consistency and coherent expression. The mapping construction unit is used to establish a hierarchical attribution mapping structure between the structured evidence summary and each supporting information, and between each supporting information and the corresponding original retrieval document; The adaptive termination evaluator module determines whether the currently verified information to be evaluated meets the preset sufficiency conditions, including: Based on at least two of the following parameters—the number of query questions corresponding to additional retrieval needs, the similarity between newly generated query questions and existing query questions, and the prediction confidence—a dynamic weight fusion algorithm is used to calculate the sufficiency score of the information to be evaluated. The calculation formula is as follows: ， Where I is the information sufficiency marker, α is the prediction confidence level corresponding to the information sufficiency label I, where α is the weighting coefficient. This is a quantitative value for the information gap calculated based on the number of additional search questions. , where n is the number of questions that need to be additionally retrieved, as given by the large language model. The maximum number of problems is defined by ; S represents the similarity between newly generated problems and existing problems. The sufficiency score is compared with a preset information sufficiency score threshold to generate a judgment result for the information to be evaluated.

2. The verifiable large model retrieval enhancement generation system based on evidence chain according to claim 1, characterized in that: The heuristic question generator module receives an initial query question, combines it with associated search documents, identifies ambiguities and information gaps in the initial query, and generates a set of supplementary query questions, including: The initial query question is taken as input, and a preliminary answer associated with the initial query question is generated through model parameterized knowledge reasoning of a preset language model. Extract the keywords of the initial query question and use them as search terms for the search engine to obtain a set of search documents associated with the initial query question; Define the semantic coverage of the initial query question, perform sentence-by-sentence semantic matching and incremental information identification on each document in the search document set, and filter out discrete fragmented information in the search documents that are not in the semantic coverage but are associated with the initial query question or the preliminary answer to obtain an incremental fragmented information set; Based on the preliminary answer and the incremental fragmented information set, the information gaps in the initial query question are identified by semantic association mapping between the fragmented information and the initial query question, and candidate supplementary query questions are generated in a targeted manner based on the information gaps. All candidate supplementary query questions are semantically redundantly eliminated and information coverage is verified to form a set of structured supplementary query questions that meet the information gap filling requirements.

3. The verifiable large model retrieval enhancement generation system based on evidence chain according to claim 1, characterized in that: The trusted answer verifier module includes: The candidate answer generation unit is used to generate an initial answer containing multiple semantically independent sentences based on a single supplementary query question and its corresponding retrieval document in the supplementary query question set, and to associate each sentence with a unique identifier of the retrieval document from which its information source is located; The single-sentence verification unit is used to perform support verification on each sentence in the initial answer, in conjunction with the source document it references, to determine whether the content of each sentence can be supported by the referenced source document, and to obtain the verification result of each sentence. The quality assessment unit is used to count the number of sentences with a support judgment result of "pass" based on the verification results of all sentences, and to calculate the overall attribution accuracy of the initial answer; Initial answers with an overall attribution accuracy greater than or equal to a preset quality assessment threshold are retained as candidate answers that meet the quality requirements.

4. The verifiable large model retrieval enhancement generation system based on evidence chain according to claim 3, characterized in that: The single-sentence verification unit performs support verification on each sentence in the initial answer, in conjunction with the source document it references. It determines sentence-by-sentence whether the content of each sentence is supported by the referenced source document, obtaining the verification result for each sentence, including: Extract each statement to be verified from the initial answer, and based on its preset reference association with the search document, match the unique search source document corresponding to each statement to be verified to construct an input pair of statement to be verified and associated document; Each input pair is fed into the preset verification model one by one. The model performs deep matching of the semantics of the statement and the content of the document, and analyzes the supporting logic of the information to obtain the judgment result of whether the statement to be verified is supported by the associated document. At the same time, the judgment reason is generated, and the set of supporting sentences that can support the judgment result is extracted from the associated document. The triplet data consisting of the judgment result, the judgment reason, and the supporting sentence set is used as the verification result for each sentence.

5. The verifiable large model retrieval enhancement generation system based on evidence chain according to claim 1, characterized in that: The hierarchical attribution mapping structure includes a first mapping layer and a second mapping layer. The first mapping layer is the mapping relationship between the structured evidence summary and the supporting information, and the second mapping layer is the mapping relationship between a single sentence in the supporting information and the unique identifier of the corresponding original retrieval source document.

6. A method for enhancing the generation of verifiable large models based on evidence chains, characterized in that, include: S1: Receive the initial query question, combine it with the associated search documents, identify the ambiguous position and information gap of the initial query question, and generate a set of supplementary query questions; S2: Based on the initial query question, the set of supplementary query questions, and the corresponding search documents, generate candidate answers with citation information, and verify the information support of the candidate answers; S3: For the verified candidate answers, extract supporting information and construct a hierarchical attribution mapping structure; S4: Integrate the verified candidate answers, their corresponding extracted supporting information, and the hierarchical attribution mapping structure into information to be evaluated, and determine whether the currently verified information to be evaluated meets the preset sufficiency conditions: If the conditions are met, the generated preliminary answer and the information to be evaluated are integrated to synthesize the target answer; If the conditions are not met, return to step S1 and continue iterating to generate new supplementary query questions based on the current information gaps; The method for extracting supporting information and constructing a hierarchical attribution mapping structure for verified candidate answers is as follows: A preset support information extraction algorithm is invoked. Based on the semantic features of the verified candidate answers and the content relevance of the retrieval source documents, a minimum set of supporting sentences that can support the validity of the candidate answers is extracted from the original retrieval source documents corresponding to the verified candidate answers as supporting information. The supporting information is aggregated to form a structured evidence summary with semantic consistency and coherent expression. Establish a hierarchical attribution mapping structure between the structured evidence summary and each supporting information, and between each supporting information and the corresponding original retrieval document; Determine whether the currently verified information to be evaluated meets the preset sufficiency conditions, including: Based on at least two of the following parameters—the number of query questions corresponding to additional retrieval needs, the similarity between newly generated query questions and existing query questions, and the prediction confidence—a dynamic weight fusion algorithm is used to calculate the sufficiency score of the information to be evaluated. The calculation formula is as follows: ， Where I is the information sufficiency marker, α is the prediction confidence level corresponding to the information sufficiency label I, where α is the weighting coefficient. This is a quantitative value for the information gap calculated based on the number of additional search questions. , where n is the number of questions that need to be additionally retrieved, as given by the large language model. The maximum number of problems is defined by ; S represents the similarity between newly generated problems and existing problems. The sufficiency score is compared with a preset information sufficiency score threshold to generate a judgment result for the information to be evaluated.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the verifiable large model retrieval enhancement generation method based on evidence chain as described in claim 6.

Images