Methods and electronic devices for constructing evidence sets in enhanced retrieval generation

By using a state-aware iterative decision-making closed loop to dynamically select and stop adding evidence, the problems of redundant evidence sets and high inference costs in existing technologies are solved, resulting in a compact and sufficient evidence set and improving the quality and efficiency of the generative model's responses.

CN123115355APending Publication Date: 2026-11-17SHANGHAI MARITIME UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611528978.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-30
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing retrieval enhancement generation systems cannot dynamically perceive the status of the selected set when constructing the evidence set, resulting in high redundancy of the evidence set, high reasoning cost, and limited response quality, and cannot effectively utilize the complementarity and non-redundancy of candidate documents.

Method used

By constructing an iterative decision-making closed loop with set state awareness, and utilizing set-related evaluation features such as non-redundancy and new contribution, evidence can be dynamically selected and stopped from being added, forming a compact and sufficient evidence set.

Benefits of technology

It effectively suppresses diminishing marginal returns as the amount of evidence increases, reduces token consumption and inference latency in generative models, and improves the traceability and quality of responses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115355A_ABST
    Figure CN123115355A_ABST
Patent Text Reader

Abstract

This invention relates to the fields of artificial intelligence and natural language processing, and provides a method and electronic device for constructing an evidence set in retrieval enhancement generation. The method includes: acquiring a user question and a set of candidate evidence, and initializing the currently selected evidence set; determining set-related evaluation features, including non-redundancy and new contribution, based on the relationship between the remaining candidate evidence and the currently selected evidence set; determining a target action based on the set state in a unified action space containing candidate selection actions and stopping actions; updating the set and redetermining the evaluation features in response to a candidate selection action for iterative decision-making; and outputting the currently selected evidence set in response to a stopping action. This invention achieves iterative evidence selection and stopping control based on set state awareness, constructing a compact and sufficient evidence set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method and electronic device for constructing evidence sets in retrieval enhancement generation. Background Technology

[0002] In retrieval-enhanced generative techniques, relevant documents are typically retrieved from external knowledge bases as context input to the generative model, thereby improving the factuality and traceability of the answers. Most existing retrieval-enhanced generative systems employ a fixed-quantity truncation strategy, selecting a predetermined number of documents based on relevance scores; or a strategy that predicts a dynamic quantity based on question complexity. However, the fixed-quantity strategy struggles to adapt to the varying information requirements of different questions, easily leading to redundant noise in simple problems or the omission of crucial evidence in complex ones. While the dynamic quantity prediction strategy can adjust the total number of documents, it typically still relies on static relevance ranking for truncation, failing to continuously utilize the current state of the selected evidence set to reassess the added value of the remaining candidate documents. This approach fails to dynamically guarantee the complementarity and non-redundancy among candidate documents during the construction process, easily retaining documents with overlapping or weak correlations, thus increasing the context length and inference cost of the generative model, and potentially reducing answer quality due to interfering information. Therefore, there is an urgent need for a technical solution that can dynamically perceive and jointly decide the timing of evidence selection and cessation based on the set state, in order to construct a compact and sufficient evidence set. Summary of the Invention

[0003] Therefore, it is necessary to address the problems of existing technologies in the evidence construction process, such as the inability to continuously evaluate the value of remaining candidates based on the selected set state, resulting in high redundancy of the evidence set, high reasoning costs, and limited answer quality. This paper proposes a method and electronic device for constructing evidence sets in retrieval enhancement generation, which realizes iterative evidence selection and stopping control based on set state awareness, thereby constructing a compact and sufficient evidence set.

[0004] This invention provides a method for constructing an evidence set in retrieval enhancement generation, comprising: acquiring a user question and a candidate evidence set, and initializing the currently selected evidence set to empty; determining the set-related evaluation features of the remaining candidate evidence based on the relationship between the remaining candidate evidence and the currently selected evidence set; wherein, the remaining candidate evidence refers to evidence in the candidate evidence set other than the currently selected evidence set, and the set-related evaluation features include non-redundancy and contribution of addition, where non-redundancy characterizes the degree of semantic repetition of candidate evidence relative to the currently selected evidence set, and contribution of addition characterizes the utility increment of candidate evidence in answering the user question after being added to the currently selected evidence set; and based on the currently selected evidence set and the remaining candidate evidence... The set state, constructed with set-related evaluation features, determines the target action in a unified action space. This unified action space includes candidate selection actions for selecting remaining candidate evidence and a stop action. In response to the target action being a candidate selection action, the target candidate evidence indicated by the target action is added to the currently selected evidence set. Based on the updated currently selected evidence set, the set-related evaluation features of the remaining candidate evidence are re-determined, and the process returns to the step of determining the target action in the unified action space based on the set state constructed with the current selected evidence set, remaining candidate evidence, and set-related evaluation features. In response to the target action being a stop action, the currently selected evidence set is output as the basis for generating retrieval enhancements.

[0005] In one embodiment, the target action is determined in a unified action space based on the current set of selected evidence, the remaining candidate evidence, and the set-related evaluation features. This includes: processing the set state through a decision model to obtain the predicted scores of each candidate selection action and the stop action in the unified action space; and determining the target action from the unified action space based on the predicted scores. The predicted score of the stop action represents the sufficiency of the current set of selected evidence in answering the user's question.

[0006] In one embodiment, based on the relationship between the remaining candidate evidence and the currently selected evidence set, the set-related evaluation characteristics of the remaining candidate evidence are determined, including: for any candidate evidence among the remaining candidate evidence... Based on candidate evidence Candidate semantic vectors and the currently selected evidence set The similarity of the semantic vectors of each selected piece of evidence is used to determine candidate evidence. The non-redundancy; where, when the currently selected evidence set When empty, the non-redundancy is the preset maximum value; when the currently selected evidence set is empty... When not empty, the non-redundancy degree is negatively correlated with the maximum similarity; the difference between the first sufficiency estimate and the second sufficiency estimate is determined as candidate evidence. The new contribution; of which, the first sufficiency estimate is the value of the new contribution; This indicates that candidate evidence will be included. Add to the currently selected evidence set Later, in response to user issues The sufficiency estimate is 1, and the second sufficiency estimate is 2. , indicating the currently selected set of evidence. Addressing user issues The sufficiency estimate.

[0007] In one embodiment, the set state includes at least the following information elements: user question; currently selected evidence set; remaining candidate evidence; and set-related evaluation features of the remaining candidate evidence.

[0008] In one embodiment, obtaining the user question and candidate evidence set includes: receiving an initial candidate set output by an external retrieval system; mapping the initial candidate set to standardized records containing candidate identifiers and candidate content to obtain a candidate evidence set; wherein the mapping process does not modify the internal recall parameters or ranking parameters of the external retrieval system.

[0009] In one embodiment, receiving an initial candidate set output by an external retrieval system includes: constructing a main retrieval branch and at least one rewritten retrieval branch based on the user's question; performing retrieval operations on each retrieval branch to obtain a candidate list for each branch; and merging the candidate lists of each branch according to the candidate identifier to obtain an initial candidate set. The standardized record also includes at least one of the following: source branch set, cross-branch hit count, intra-branch sorting information, and retrieval score.

[0010] In one embodiment, the method further includes: concatenating the evidence in the currently selected evidence set according to the order of addition to generate context information; and inputting the user question and the context information into the generation model to obtain the answer result for the user question.

[0011] In one embodiment, the method is executed by a decision model, which is trained as follows: The decision model acquires the state-action trajectory corresponding to the training sample set; wherein the state-action trajectory includes the sample set state and action labels output by the teacher model, the action labels including a sample candidate selection label or a sample stop label, the sample candidate selection label indicating the sample evidence that should be added to the sample's already selected evidence set, and the sample stop label indicating the termination of the selection process; the student model is supervised and trained based on the state-action trajectory, and the trained student model is used as the decision model; wherein the sample set state includes the sample question, the sample's already selected evidence set, the remaining sample candidate evidence, and the relevant evaluation features of the sample set.

[0012] In one embodiment, determining the new contribution of the set of evaluation features related to the remaining candidate evidence includes: for any candidate evidence among the remaining candidate evidence, constructing a set of hypothetical evidence after adding the candidate evidence to the currently selected evidence set; constructing a hypothetical set state based on the user question, the hypothetical evidence set, other candidate evidence besides the candidate evidence among the remaining candidate evidence, and the non-redundancy of other candidate evidence relative to the hypothetical evidence set; processing the hypothetical set state through a decision model to obtain a first predicted score for stopping the action under the hypothetical set state; processing the set state through a decision model to obtain a second predicted score for stopping the action under the set state; and determining the difference between the first predicted score and the second predicted score as the new contribution of the candidate evidence.

[0013] In one embodiment, the decision model is trained as follows: The state-action trajectory formed by the teacher model in each round of decision-making for the sample problem is obtained; wherein, the input of the teacher model in each round of decision-making includes the sample problem, the set of selected evidence, the remaining candidate evidence, and the relevant evaluation features of the sample set, and the input of the teacher model does not include the gold standard answer and supporting evidence labels of the sample problem; the action labels of each round in the state-action trajectory include a sample candidate selection label or a sample stop label, the sample candidate selection label indicating the sample evidence that should be added to the set of selected evidence, and the sample stop label indicating the termination of the selection process; the student model is supervised and trained using the sample set state of each round in the state-action trajectory as input and the corresponding round action labels as supervision signals, and the trained student model is used as the decision model; wherein, the sample set state includes the sample problem, the set of selected evidence, the remaining candidate evidence, and the relevant evaluation features of the sample set.

[0014] In addition, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in any of the above embodiments.

[0015] In addition, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above embodiments.

[0016] The aforementioned evidence set construction method and electronic device in retrieval enhancement generation model the evidence construction process as an iterative decision-making closed loop driven by the set state. It utilizes set-related evaluation features, including non-redundancy and new contribution, to quantify the dynamic value of candidate evidence relative to the currently selected set, rather than relying solely on static relevance ranking. This mechanism allows each round of decision-making to perceive changes in the state of the selected set, thereby accurately identifying evidence that can fill information gaps and does not duplicate existing content.

[0017] Furthermore, by placing candidate selection actions and stopping actions within the same unified action space for joint decision-making, the order of evidence selection and the timing of stopping are determined by the same strategy competing under the same probability distribution. This avoids the fragmentation caused by pre-setting a fixed number or separately predicting a dynamic number, allowing the size of the evidence set to be naturally determined by the sufficiency of the content, thereby compressing the context length while maintaining the quality of the response.

[0018] Furthermore, by triggering a set state update and feature recalculation mechanism after each new piece of evidence is selected, a feedback loop of selection, updating, and revaluation is formed. This loop can effectively suppress the diminishing marginal returns that occur as the number of pieces of evidence increases, promptly eliminate redundant or interfering documents with reduced value in subsequent rounds, reduce token consumption and inference latency of the generative model, and improve the traceability of the final answer. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram comparing the fixed Top-k retrieval method and the information gain-driven evidence construction method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall architecture for constructing compact and sufficient evidence driven by information gain according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the collection state-aware evidence selection and cessation closed-loop process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the teacher trajectory generation, supervised distillation, and plug-in inference deployment process according to an embodiment of the present invention; Figure 5 This is a cumulative information gain curve of different methods in the embodiments of the present invention as a function of the amount of evidence; Figure 6 This is a proportional diagram showing the functional composition of documents retrieved by different methods in embodiments of the present invention; Figure 7 This is a cumulative information gain curve after the decision model of this invention is integrated into an external retrieval system. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0023] The following is combined Figures 1-7 This invention describes a method for constructing evidence sets and an electronic device for enhancing retrieval.

[0024] In one embodiment, a method for constructing an evidence set in retrieval enhancement generation is provided, which constructs a compact and sufficient evidence set through a set-state-aware iterative decision-making closed loop.

[0025] Reference Figure 1 The fixed Top-k retrieval method uniformly extracts documents based on single-time relevance ranking, which easily retains documents with overlapping content and interfering documents irrelevant to the answer. The method in this embodiment represents the selection of candidate evidence as a closed-loop decision-making process driven by the set state: the non-redundancy and contribution of each candidate are calculated based on the currently selected set, the candidate selection action and the stop action are jointly compared in a unified action space, and the set state and remaining candidate features are updated after each selection, thereby directly constructing a complementary evidence set that supports the answer; the size of the evidence set is the result of the closed-loop selection process, rather than a pre-set control parameter.

[0026] like Figure 2 and Figure 3 As shown, the specific execution steps of this method are as follows: Step S110: Obtain the user's question and the set of candidate evidence, and initialize the currently selected evidence set to empty.

[0027] Specifically, the system receives natural language questions or task instructions from the user as user questions, and retrieves an initial set of candidate documents from external retrieval systems or local knowledge bases as a candidate evidence set. At the initial moment of constructing the evidence set, the currently selected evidence set... Set as empty set This indicates that no evidence has yet been included. This step establishes the initial state for iterative decision-making, making subsequent evidence selection dependent on the dynamic evolution of the set of states, rather than a predetermined fixed number.

[0028] Step S120: Based on the relationship between the remaining candidate evidence and the currently selected evidence set, determine the set-related evaluation features of the remaining candidate evidence; wherein, the remaining candidate evidence is the evidence in the candidate evidence set other than the currently selected evidence set, and the set-related evaluation features include non-redundancy and new contribution. Non-redundancy represents the degree of semantic repetition of the candidate evidence relative to the currently selected evidence set, and new contribution represents the utility increment of the candidate evidence in answering the user's question after it is added to the currently selected evidence set.

[0029] In this step, the evaluation features are not calculated based on the static correlation between candidate evidence and the user's question, but rather on a dynamic measurement conditioned on the currently selected evidence set. As the content of the selected set increases, the evaluation feature values ​​of the same candidate evidence will change, thus reflecting its selection value in different set contexts.

[0030] In one implementation, for any candidate evidence among the remaining candidate evidence... Based on candidate evidence Candidate semantic vectors and the currently selected evidence set The similarity of the semantic vectors of each selected piece of evidence is used to determine candidate evidence. The non-redundancy. The specific calculation can be expressed using the following formula: , in, Indicate candidate evidence Compared to the current set of selected evidence Non-redundancy, and They represent candidate evidence respectively. and selected evidence The semantic vector after L2 normalization This represents the cosine similarity function. When the selected set is empty, the non-redundancy score takes the preset maximum value of 1; when the selected set is not empty, the non-redundancy score is negatively correlated with the maximum similarity score, and its value is truncated in the interval [0,1]. This index quantifies the degree of semantic repetition of candidate evidence relative to the selected set. The higher the value, the more novel information the candidate evidence contains and the less overlap with the selected content.

[0031] Meanwhile, the difference between the first sufficiency estimate and the second sufficiency estimate is determined as candidate evidence. The newly added contribution. Its mathematical definition is as follows: , in, Indicate candidate evidence In the New contributions in rounds of decision-making The first sufficiency estimate indicates that candidate evidence is included. Add to the currently selected evidence set Later, in response to user issues The sufficiency estimate; The second sufficiency estimate represents the current set of selected evidence. Addressing user issues The sufficiency estimate is the marginal information gain, which directly measures the incremental utility of candidate evidence in the response task. By combining non-redundancy and contribution, evidence can be screened from both complementarity and effectiveness dimensions, avoiding the accumulation of redundancy caused by relying solely on relevance ranking.

[0032] Step S130: Based on the current set of selected evidence, the remaining candidate evidence, and the set state constructed from the set-related evaluation features, determine the target action in the unified action space; wherein, the unified action space includes candidate selection actions for selecting the remaining candidate evidence and stop actions.

[0033] This step places evidence selection and stopping judgment within the same decision-making framework. Stopping is no longer regarded as a post-threshold judgment or a preset quantity cutoff that is independent of selection, but rather as an option that competes alongside specific candidate selection actions.

[0034] In one implementation, the set state includes at least the following information elements: the user question; the currently selected evidence set; the remaining candidate evidence; and the set-related evaluation features of the remaining candidate evidence. Specifically, these elements can be serialized into structured text or vector representations as input to the decision model. For example, the user question, the list of selected evidence and its features, the list of remaining candidates and their non-redundancy and new contribution can be concatenated into a cue word template or encoded into a state vector, enabling the decision model to obtain the current evidence construction progress and information gaps.

[0035] Furthermore, the set of states is processed through a decision model to obtain the predicted scores of each candidate action and the stop action in the unified action space. Based on these predicted scores, the target action is determined from the unified action space. The predicted score of the stop action characterizes the sufficiency of the currently selected evidence set in answering the user's question. The decision model can be a trained language model, classifier, or policy network. In each round of decision-making, the model output covers the probability distribution or scalar score of all remaining candidates and the stop action. Candidate actions compete with the stop action under the same probability distribution, and the one with the highest score is determined as the target action. The predicted score of the stop action also characterizes the sufficiency of the currently selected evidence set in answering the user's question. When this score exceeds the scores of all candidate actions, it indicates that the marginal information gain from introducing new evidence is less than the completeness of the current set, and the system terminates the selection process.

[0036] The set state of the decision in round t can be represented as: Where q represents the user question. This is the current set of selected evidence. For the remaining candidate evidence, Let the set of remaining candidate evidence be related to the evaluation features; let the stopping action be denoted as . The unified action space can be represented as The predicted score of the stopping action is used as a sufficiency estimate of the current set, i.e. In other words, the sufficiency estimate is numerically equal to the stop action prediction score obtained by the decision model after processing the set of states. Based on this, the additional contribution can be expressed as the difference in sufficiency estimates before and after the addition of candidate evidence: In the first implementation, the difference is explicitly estimated for each candidate and the candidate with the largest gain is selected; in the second implementation, the decision model directly outputs the next action with the highest predicted score in a unified action space, without enumerating the hypothesis set one by one; both implementations use the same definition for the new contribution.

[0037] Step S140: In response to the target action being a candidate selection action, add the target candidate evidence indicated by the target action to the current selected evidence set, redetermine the set-related evaluation features of the remaining candidate evidence based on the updated current selected evidence set, and return to the execution of the set state constructed based on the current selected evidence set, the remaining candidate evidence, and the set-related evaluation features, and determine the target action in the unified action space.

[0038] When the decision result is to select a candidate piece of evidence, that evidence is added to the selected set, and the set state changes accordingly. At this point, the system recalculates the non-redundancy and contribution of each remaining candidate piece of evidence based on the updated selected set. This is because the addition of new evidence may cover certain information gaps, causing other candidates that originally had high contribution to become redundant or less valuable. Through this feedback loop of "selection-update-recalculation," the evidence construction process makes decisions based on the latest set state, suppressing the accumulation of low-yield evidence; this update can be represented as... ,in Candidate evidence of the target indicated by the target action.

[0039] In step S150, in response to the target action being a stop action, the currently selected evidence set is output as the basis for generating retrieval enhancement.

[0040] When the "stop" action wins in the competition, it indicates that the current set of selected evidence has reached a sufficient state to support the answer, or that the remaining candidates cannot provide effective supplementation. At this point, the system terminates the iteration loop and outputs the current set of selected evidence as the final result. The size of this set is determined by the sufficiency of the content, rather than being pre-set, thus adapting to problems of varying complexity. Furthermore, when the set of remaining candidate evidence is empty, the unified action space contains only the "stop" action, and the system still outputs the current set of selected evidence. Optionally, an upper limit on the number of decision rounds or an upper limit on the input length can be set as an exception fallback; this constraint does not replace the joint decision driven by the set state.

[0041] In a further implementation, the evidence in the currently selected evidence set is concatenated according to the order in which it was added to generate context information. The user question and the context information are then input into the generative model to obtain an answer to the user question. Since the evidence is added sequentially in descending order of contribution, this order inherently implies a ranking of importance. Concatenating the generated context information in this order helps the generative model prioritize high-value evidence and mitigate the problem of intermediate forgetting in long contexts. Figure 2 and Figure 3 As shown, the above steps S110 to S150 constitute a complete information gain-driven evidence construction closed loop, completing the transformation from a candidate evidence set to a compact and sufficient evidence set.

[0042] Figure 5 and Figure 6 The data trends can verify the technical effectiveness of the method in this embodiment. For example... Figure 5As shown, the horizontal axis of the graph represents the number of retrieved documents k, and the vertical axis represents the average cumulative information gain (F1 score or accuracy). The graph contains eight subgraphs, four at the top and four at the bottom, which correspond to the Natural Questions dataset (NQ), the HotpotQA dataset, the 2WikiMultiHopQA dataset, and the MultiHopRAG dataset, respectively. The top row compares the method of this embodiment with fixed-budget methods, namely Information Gain Retrieval Enhancement Generation-Supervised Fine-Tuning (IGRAG-SFT, i.e., the decision model obtained through supervised training in this embodiment), Information Gain Retrieval Enhancement Generation-Teacher Model (IGRAG-Teacher), Standard Retrieval Enhancement Generation (Standard RAG), Rerank Retrieval Enhancement Generation (Rerank RAG), and Large Model Ranking Retrieval Enhancement Generation (RankLLMRAG). The bottom row further compares it with Standard Retrieval Enhancement Generation (Standard RAG) and dynamic retrieval methods such as Look-Ahead Active Retrieval (FLARE), Retrieval Compression (RECOMP), Interleaved Retrieval Chain Reasoning (IRCoT), and Iter-RetGen. On various datasets, the curves of IGRAG-SFT and IGRAG-Teacher both rise rapidly in the first 1 to 2 documents and then tend to flatten out, while the curves of the other comparison methods rise more slowly or fluctuate more, which is consistent with the mechanism of this embodiment of "prioritizing the inclusion of evidence with high contribution and triggering a stopping action when the gain is saturated".

[0043] like Figure 6 As shown in the figure, the vertical axis represents the percentage of retrieved documents, and the horizontal axis lists seven methods: Information Gain Retrieval Enhancement Generation - Supervised Fine-tuning, Standard Retrieval Enhancement Generation, Interleaved Retrieval Chain Reasoning, Proactive Lookout Retrieval, Self-RAG Retrieval Enhancement Generation, and Retrieval Compression and Iterative Retrieval Generation. The selected documents are statistically categorized into five functional percentages: useful, bridging, redundant, interfering, and irrelevant. In this embodiment, the method selects documents where usefulness accounts for 35.4%, bridging for 28.1%, totaling 63.5%, and redundancy for only 3.2%. In contrast, Standard Retrieval Enhancement Generation's usefulness, bridging, and redundancy are 17.6%, 12.5%, and 17.9%, respectively; Interleaved Retrieval Chain Reasoning's are 25.8%, 14.5%, and 12.8%, respectively; and the remaining comparative methods all have a usefulness percentage no higher than 18% and a redundancy percentage no lower than 13%. This indicates that by jointly evaluating the non-redundancy and contribution of the evidence set based on set state awareness, and by unifying the selection and stopping decisions, it is possible to reduce redundant and interfering evidence while improving the information density and complementarity of the evidence set, thereby providing input for the generative model.

[0044] Table 1 compares the response performance and evidence efficiency of each method on four datasets. The evaluation metrics are EM (exact match), F1 (F1 score), and Acc (accuracy), where ↓ indicates that the smaller the better and ↑ indicates that the larger the better.

[0045] As shown in Table 1, the fixed budget method did not consistently achieve stable returns as the number of documents increased from 3 to 10. The proposed method (teacher model) and the proposed method (student model) achieved the best or near-best overall response performance with an average of 3.13 and 3.36 documents, respectively. The proposed method (student model) achieved an F1 score of 30.65 on the two-dimensional multi-hop question answering dataset and an accuracy of 62.30 on the multi-hop retrieval augmented generation dataset. The proposed method (teacher model) achieved the highest F1 scores of 53.58 and 45.97 on the natural question dataset and the hot topic question answering dataset, respectively. This indicates that dynamic selection and sufficiency stopping can simultaneously improve evidence quality and contextual efficiency.

[0046] To further analyze the impact of evidence selection on the performance of traditional ranking retrieval, this embodiment independently evaluates retrieval quality before answer generation. The evaluation metrics are Precision@5 (top 5 precision), Recall@5 (top 5 recall), NDCG@5 (top 5 normalized depreciation cumulative gain), and MRR@5 (top 5 average reciprocal rank). Table 2 lists the complete retrieval quality comparison results on the hot question answering dataset.

[0047] As shown in Table 2, on the hot topic question-answering dataset, the methods of this invention (teacher model) and the methods of this invention (student model) outperform other comparative methods in all four retrieval metrics. Compared with the results of non-specific embodiments with the strongest performance in each metric, the methods of this invention (student model) improved Precision@5 from 45.12 to 50.86, Recall@5 from 55.92 to 59.80, NDCG@5 from 48.31 to 53.60, and MRR@5 from 46.62 to 52.09, representing increases of 5.74, 3.88, 5.29, and 5.47 percentage points, respectively. This indicates that this embodiment can more centrally arrange supporting evidence at the beginning of the evidence sequence. The differences between the methods of this invention (student model) and the methods of this invention (teacher model) in the four metrics are only 0.35, 0.50, 0.56, and 0.86 percentage points, respectively, indicating that supervised distillation effectively preserves the evidence priority judgment ability of the teacher model.

[0048] Table 3 lists the ablation results of the main components of this embodiment to verify the function of each component.

[0049] The results above indicate that multi-query hybrid pooling primarily improves complementary candidate coverage and reduces the number of documents required to achieve sufficiency; joint selection and stopping decision-making (i.e., the decision model closed loop in this embodiment) is the core of dynamic evidence construction; and teacher trajectory distillation plays an important role in enabling the lightweight student model to learn stable selection and stopping behaviors.

[0050] In one embodiment, the process of acquiring and standardizing the candidate evidence set is further detailed. This process, as a preliminary step in the evidence construction loop, aims to provide a unified and structured data foundation for subsequent set-state awareness decisions, while maintaining decoupling from the upstream retrieval system.

[0051] Step S210: Receive the initial candidate set output by the external retrieval system.

[0052] Specifically, the method of this invention can be deployed as a standalone plug-in or middleware module, receiving raw search results from any external retrieval system through a standardized application programming interface (API) or data pipeline. This external retrieval system can be a sparse search engine, a dense vector search engine, a hybrid search engine, a knowledge graph search engine, or any combination thereof. During the receiving process, only the candidate document list output by the external system and its accompanying metadata are read, without any modification or intervention to the recall strategy, index structure, ranking algorithm, or model parameters within the external retrieval system. This non-intrusive access method allows the evidence construction method of this invention to adapt to different retrieval backends without requiring customized development or retraining for specific retrieval systems, thereby reducing system integration costs and maintenance complexity, and providing a universal interface foundation for subsequent migration between different retrieval architectures.

[0053] Step S220: Construct a main search branch and at least one rewritten search branch based on the user's question; perform search operations on each search branch to obtain a candidate list for each branch; and merge the candidate lists of each branch according to the candidate identifier to obtain an initial candidate set.

[0054] In a preferred embodiment of this step, to overcome the potential for insufficient recall or semantic bias caused by a single query expression, the system first retains the original question input by the user as the main retrieval branch, and then uses a query rewriting model to generate one or more semantically equivalent but differently expressed rewritten retrieval branches. Let the original question be q0, and the rewritten branches be q1,...,q_m, then... Where g is the query rewriting model and m is the upper limit of the number of branches; each branch returns at most M candidate documents from the corpus D via the retrieval machine R: The candidate lists of each branch are merged after deduplication based on candidate identifiers to obtain the initial candidate set: For example, for complex multi-hop problems, rewriting branches can focus on different sub-problem entities or relational paths. Subsequently, an external retrieval system is used to perform searches on the main branch and each rewriting branch, obtaining a list of candidate documents for each branch. During the merging phase, the system deduplicates the results of each branch based on stable candidate identifiers (such as unique document IDs or content hash values). For the same candidate document that is hit by multiple branches, duplicates are not simply discarded; instead, the hit information across different branches is aggregated. For example... Figure 2 As shown in the figure (the “multi-query hybrid evidence pool” corresponds to the initial candidate set in this embodiment), this multi-branch fusion mechanism expands the coverage of candidate evidence and reduces single rewriting bias through cross-branch consistency verification, thereby alleviating the problem of missing key evidence caused by the limitation of a single search perspective.

[0055] Step S230: Map the initial candidate set to standardized records containing candidate identifiers and candidate content to obtain a candidate evidence set; wherein, the mapping process does not modify the internal recall parameters or ranking parameters of the external retrieval system, and the standardized records also include at least one of the following: source branch set, cross-branch hit count, intra-branch ranking information, and retrieval score.

[0056] Specifically, the system uniformly transforms heterogeneous candidate data from different retrieval branches or external systems into a predefined standardized data structure. This standardized record contains at least candidate identifiers for uniquely identifying evidence and candidate content text for subsequent semantic analysis. Furthermore, the record incorporates structured metadata fields reflecting the evidence retrieval context. For example, the source branch set field records which retrieval branches the candidate evidence was hit by, which can be used to determine the semantic robustness of the evidence; the cross-branch hit number quantifies the frequency of evidence recurrence under multi-perspective retrieval, with evidence that hits at high frequencies typically having higher reliability; and the intra-branch ranking information and retrieval score fields retain the original retrieval tool's preliminary assessment of the evidence's relevance. These metadata fields serve as input signals for calculating set-related evaluation features in subsequent steps. For example, the cross-branch hit count can be used as a priori weight to assist in correcting the calculation of non-redundancy, or as an auxiliary factor to distinguish evidence quality in the estimation of new contribution. By transforming the original retrieval results into standardized records carrying structured signals, this embodiment provides data support for subsequent set-state-aware decision-making without relying on the internal implementation of a specific retrieval tool, enabling the decision model to comprehensively utilize content semantics and retrieval statistical features to identify complementary and non-redundant evidence.

[0057] In one implementation, the candidate evidence set can be initially globally sorted sequentially by descending cross-branch hit count, descending maximum retrieval score, and ascending optimal branch ranking. This initial sorting is used to organize candidates or control input length and does not replace subsequent set-level selection. The maximum number of branches, m, can be 3 and can be adjusted according to task complexity or computational budget. In addition to the aforementioned cosine similarity-based calculation method, non-redundancy can also be calculated using alternative methods such as maximum marginal relevance, deduplication models, semantic implication relationships, fact triple overlap rate, or topic coverage difference.

[0058] In one implementation, the aforementioned method can also access an external retrieval system capable of outputting candidate documents or candidate evidence units, and this access is positioned between the candidate result output and subsequent reasoning or answer generation. During access, the internal recall and ranking parameters of the external retrieval system are not modified; instead, its output candidates are mapped to the aforementioned standardized records, and set state construction, candidate feature recalculation, and joint selection and stopping decisions are added to the candidate output. The candidate sequence obtained by any external retrieval system is denoted as... Based on the aforementioned user question q and the currently selected set of evidence... and remaining candidates As input, and optionally as the current inference state of the external system. Input, output the next action ,in To stop the action; when a candidate piece of evidence is output, add it to the current set of selected evidence and recalculate the relevant features of the remaining candidates; when the stop action is output, feed the current set back to the external system for subsequent reasoning or generation steps. For a one-time retrieval system, this step can be omitted. For iterative retrieval systems using interleaved retrieval and chain reasoning, It can include the current intermediate inference results.

[0059] Figure 7The curve trend verifies the effectiveness of the aforementioned access method. The figure shows the average cumulative information gain curve (F1 score), with the horizontal axis representing the number of retrievals. The two sub-graphs correspond to the multi-hop question-answering dataset (HotpotQA) and the two-dimensional dataset (2WikiMultiHopQA), respectively. The curves in the figure include information gain retrieval enhancement generation-supervised fine-tuning (i.e., the student model of the present invention), interleaved retrieval chain reasoning after integrating the aforementioned methods (labeled as "iterative retrieval chain reasoning-stopper" in the figure), and interleaved retrieval chain reasoning. Taking integration into interleaved retrieval chain reasoning as an example, the latter is responsible for expanding the search scope. The aforementioned method performs set-related feature updates, complementary evidence selection, and stopping control at its candidate output end. As shown in the figure, on both datasets, the integrated curve rises rapidly in the first 1 to 2 pieces of evidence, and is generally higher than the unintegrated interleaved retrieval chain reasoning curve. After a small number of pieces of evidence, it tends to plateau, indicating that the aforementioned method prioritizes retaining highly useful evidence and terminates ineffective expansion in a timely manner after the evidence becomes sufficient.

[0060] Table 4 lists the results of integrating the joint decision of selection and stopping into interleaved retrieval chain reasoning, where EM represents exact match and F1 represents F1 score.

[0061] As shown in Table 4, after plugging in the code, on HotpotQA, the integrated stopper improved EM from 29.10 to 36.50, F1 from 39.02 to 48.73, and the average number of pieces of evidence decreased from 15.50 to 7.29, a reduction of approximately 53%. On 2WikiMultiHopQA, EM improved from 19.20 to 23.27, F1 from 27.75 to 33.92, and the average number of pieces of evidence decreased from 9.17 to 5.41, a reduction of approximately 41%. This indicates that the aforementioned plug-in access method, while adapting to external iterative retrieval systems, reduces the scale of evidence entering subsequent inference chains and generators, and improves the quality of responses.

[0062] In one embodiment, a specific implementation of determining the incremental contribution of the set of relevant evaluation features for the remaining candidate evidence is further detailed. The incremental contribution is the increase in utility for answering the user's question after the addition of candidate evidence. This embodiment provides an explicit calculation method based on the difference in hypothetical state scores, which quantifies this increment by constructing a virtual future decision state, thereby providing a calculable numerical basis for evidence selection.

[0063] Step S310: For any candidate evidence among the remaining candidate evidence, construct a set of hypothetical evidence after adding the candidate evidence to the current set of selected evidence.

[0064] Specifically, in each round of decision-making, the system iterates through every candidate piece of evidence in the current set of remaining candidate evidence. And consider it as the object to be evaluated. To measure this candidate evidence... For each new contribution, the system creates a temporary set of hypothesis evidence in memory. ,in This represents the current set of truly selected evidence. This set of evidence is assumed not to be a real, persistent storage update, but rather a logical snapshot existing only in the current computational context, used to simulate the scenario where candidate evidence... The new evidence combination formed after inclusion in the set. This construction method enables the system to independently and in parallel pre-evaluate the potential contribution of each candidate piece of evidence without changing the actual retrieval state.

[0065] Step S320: Based on the user question, the hypothesis evidence set, the remaining candidate evidence (excluding the candidate evidence), and the non-redundancy of the other candidate evidence relative to the hypothesis evidence set, construct the hypothesis set state.

[0066] In this step, the system utilizes the set of hypothesis evidence generated in step S310. To construct a complete set of hypothesis states This status includes user issues. and set of hypothetical evidence It itself, and includes the updated set of remaining candidate evidence and its features. Due to the candidate evidence... The remaining candidate evidence set has been moved into the hypothesis-selected set. At the same time, the non-redundancy of the remaining candidate evidence relative to the set needs to be based on the new set of hypothetical evidence. A recalculation is required. This is due to the newly added candidate evidence. This may cover certain semantic spaces, causing changes in the degree of repetition of other remaining candidate evidence relative to the new set. For example, if candidate evidence... With a certain remaining candidate evidence If the semantics are highly similar, then in the hypothetical state... Down, The non-redundancy will decrease. Through this dynamic recalculation, it is assumed that the set state reflects the introduced candidate evidence. The changes in the post-evidence space provide contextual input for subsequent adequacy assessments.

[0067] Step S330: Process the state of the hypothesis set through the decision model to obtain the first prediction score of the stop action under the state of the hypothesis set.

[0068] Specifically, the constructed set of hypothetical states The input is fed into a pre-trained decision model. The decision model infers from this state and outputs the predicted scores or probability distributions of each action in a unified action space. The predicted score corresponding to the stop action is extracted from this and denoted as the first predicted score. This score represents the score assuming candidate evidence has already been selected. In this case, the decision model assesses the sufficiency of the current evidence set in answering the user's question. That is, the score reflects the adequacy of the included candidate evidence. A quantitative estimate of the sufficiency of the subsequent evidence set in answering the user's question. A higher first prediction score indicates a greater number of candidate evidence pieces included. Subsequently, the closer the evidence set is to a sufficient state, i.e., candidate evidence... The greater the contribution to filling information gaps.

[0069] Step S340: Process the set state through the decision model to obtain the second prediction score of the stop action under the set state.

[0070] At the same time, or before that, the system also needs to obtain the baseline sufficiency score under the current real-world state. This involves determining the current real-world set state. (Includes user issues) The current set of selected evidence Original remaining candidate evidence and its relative to (Non-redundancy) is input into the same decision model, and the obtained stop action prediction score is used as the second prediction score. This score represents the score in the absence of candidate evidence. At that time, the sufficiency level of the current evidence set itself. In actual engineering implementation, because the current true set state changes within the same round of decision iteration... Since all candidate evidence to be evaluated is the same, the second prediction score only needs to be calculated once and can be cached and reused, eliminating the need for repeated reasoning for each candidate evidence and thus reducing the additional computational overhead of explicit calculation.

[0071] Step S350: The difference between the first predicted score and the second predicted score is determined as the new contribution of the candidate evidence.

[0072] The specific calculation formula is as follows: , in, Candidate evidence The new contribution. This difference directly quantifies the introduction of candidate evidence. The extent to which this improvement in sufficiency is achieved. If the difference is positive and large, it indicates that the candidate evidence... This can effectively improve the completion rate of the answering task; if the difference is close to zero or negative, it indicates that the candidate evidence... This approach offers no substantial help to the current task and may even introduce noise. This explicit calculation method transforms the aforementioned new contribution into a calculable numerical indicator, providing an interpretable and explicit numerical basis for the new contribution. Simultaneously, the method clarifies the generation logic of the new contribution through a complete data flow path of "constructing the hypothetical state - predicting the score - calculating the difference." Compared to methods that rely solely on static correlation scoring, this explicit calculation mechanism can identify complementary and redundant relationships between evidence, ensuring that evidence selection decisions serve to improve the quality of the response.

[0073] In one embodiment, the training method of the decision model is further detailed. Steps S110 to S150 described how the decision model performs evidence selection and stopping control based on the set state during the runtime phase. This embodiment illustrates how the decision model is obtained through supervised distillation during the training phase. Specifically, the decision model is trained as follows: The state-action trajectory corresponding to the training sample set is obtained; wherein, the state-action trajectory includes the sample set state and the action labels output by the teacher model, the action labels include sample candidate selection labels or sample stop labels, the sample candidate selection label indicates the sample evidence that should be added to the sample's already selected evidence set, and the sample stop label indicates the termination of the selection process; supervised training of the student model is performed based on the state-action trajectory, and the trained student model is used as the decision model; wherein, the sample set state includes the sample question, the sample's already selected evidence set, the remaining sample candidate evidence, and the relevant evaluation features of the sample set.

[0074] like Figure 4 As shown, this training process aims to transfer the stepwise decision-making capabilities of a large teacher model under complex set states to a student model with smaller parameters and higher inference efficiency, thereby alleviating the high latency and high cost problems caused by repeatedly calling large models during online deployment. Unlike traditional question-answer pair fine-tuning, the training data in this embodiment is not a simple "question-answer" pair, but rather a "state-action" sequence containing the complete decision-making chain, enabling the student model to learn dynamic strategies of when to select evidence and when to stop selecting.

[0075] In a preferred embodiment, the process of acquiring the state-action trajectory formed by the teacher model in each round of decision-making for the sample question has leakage prevention constraints. Specifically, the input of the teacher model in each round of decision-making includes the sample question, the set of selected evidence for the sample, the remaining candidate evidence for the sample, and the relevant evaluation features of the sample set. The input of the teacher model does not include the gold standard answer to the sample question or supporting evidence labels. If the teacher model is allowed direct access to the gold standard answer or supporting evidence labels, it is prone to degenerate into selecting documents containing answer keywords based solely on lexical overlap or semantic similarity, failing to learn to evaluate the logical sufficiency of the evidence set for answering the question. By shielding this posterior information, the teacher model is forced to reason only based on the currently observable evidence state, ensuring that the generated trajectory reflects the reasoning process from information gap identification to evidence completion. This leakage-free trajectory data enables the student model to learn set state awareness rather than overfitting to specific dataset labels, thereby helping to reduce overfitting of the student model to specific dataset labels.

[0076] Step S410: Obtain the state and action trajectory formed by the teacher model in each round of decision-making for the sample problem.

[0077] In this step, the system iterates through each sample question in the training sample set, driving the teacher model to simulate the aforementioned evidence construction loop. In each iteration, the teacher model receives the current sample set state as input, and outputs an action after internal reasoning. If the output is a sample candidate selection label, the sample evidence indicated by that label is added to the sample's selected evidence set, and the remaining candidate evidence and its set-related evaluation features are updated, proceeding to the next round of decision-making; if the output is a sample stop label, the trajectory generation of the current sample is terminated. Throughout the process, the teacher model makes judgments based on the sample question and the dynamically evolving evidence state, without relying on any externally provided correct answer hints. For example, when dealing with a problem requiring three-hop reasoning, the teacher model might sequentially select the document defining entity A, the document connecting entity A and entity B, and the document describing the attributes of entity B, finally outputting a stop action when confirming that the information is complete. This series of coherent actions and their corresponding intermediate states together constitute a complete state-action trajectory.

[0078] Step S420: Construct the data structure for the state-action trajectory.

[0079] Specifically, each state-action trajectory consists of multiple time steps, each containing a pair of sample set states and action labels. The sample set state contains four dimensions of information: the sample question, the set of selected evidence, the remaining candidate evidence, and relevant evaluation features of the sample set. The sequential representation of these four elements is consistent with the runtime phase to align the data distribution for training and inference. The action label indicates the action to be taken in the current state. For the sample candidate selection label, its content is a unique identifier for the sample evidence; for the sample stop label, its content is a predefined stop instruction identifier. This structured trajectory data records the final selection result and preserves the contextual dependencies in the decision-making process, providing supervisory signals for the student model. For example, when the set of selected evidence already contains all the key facts required for the answer, even if there are still highly relevant documents among the remaining candidates, the action label should still be the sample stop label, thus teaching the student model to identify sufficiency boundaries and avoid over-searching.

[0080] Step S430: Supervised training of the student model is performed based on the state-action trajectory, and the trained student model is used as the decision model.

[0081] In this step, supervised fine-tuning is used to optimize the student model. Specifically, using the sample set states of each round in the state-action trajectory as input and the action labels of the corresponding rounds as supervision signals, the loss value between the student model's predicted action distribution and the teacher model's actual action labels is calculated. The loss function typically uses cross-entropy loss, which aims to minimize the probability bias of the student model outputting the correct action in a given state. It can be expressed as: ,in For the sample set state of each round, These are the action labels for the corresponding rounds. During training, the student model continuously adjusts its parameters through backpropagation, gradually approximating the decision-making behavior of the teacher model. Because the student model has far fewer parameters than the teacher model, its computational overhead and memory usage during the inference phase are reduced. Furthermore, through the aforementioned distillation process, the teacher model's capabilities in set state perception, complementary evidence recognition, and dynamic stopping judgment are obtained. Figure 4 As shown, the trained student model can be deployed as a lightweight decision-making model in online services, replacing the teacher model to perform real-time evidence selection and stop control tasks, reducing system response latency and resource consumption while maintaining response quality. It should be understood that although this embodiment uses supervised fine-tuning as an example, in other implementations, imitation learning or other methods can be used to train the student model using state-action trajectories, as long as the core objective is to reproduce the teacher's aggregate state decision-making behavior under leak-free conditions.

[0082] In one implementation, the query rewriting model uses Qwen-Turbo, generating a maximum of three rewriting branches in addition to the original question. The original branch and each rewriting branch retrieve Top-6 documents, with a dense search engine using BAAI / bge-large-en-v1.5. The teacher model uses Qwen-Plus, and the student model uses Qwen2.5-7B-Instruct. The teacher model generates approximately 115K state-action trajectories based on training set questions from HotpotQA, 2WikiMultiHopQA, and MuSiQue. The answer generator can use Llama3.1-8B-Instruct. The above models and quantities are merely examples and do not constitute a limitation on the scope of protection.

[0083] In one embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned evidence set construction method in retrieval enhancement generation. Specifically, the electronic device may be a physical entity with data processing capabilities, such as a server, personal computer, mobile terminal, edge computing node, or dedicated artificial intelligence acceleration device. The memory may include volatile memory, such as random access memory; or it may include non-volatile memory, such as flash memory, hard disk, or solid-state drive. The processor may be a general-purpose processor, including a central processing unit, network processor, etc., or it may be a digital signal processor, application-specific integrated circuit, field-programmable gate array, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. When the processor reads and executes the computer program stored in the memory, the electronic device is configured to perform a series of operation steps, including obtaining the user question and candidate evidence set, determining the relevant evaluation features of the set, constructing the set state and determining the target action in a unified action space, updating the set and recalculating features in response to candidate selection actions, and outputting the evidence set in response to a stop action. Through the collaborative work of hardware resources and software programs, this electronic device can transform iterative decision-making logic based on set state awareness into actual machine processing behavior, thereby supporting the construction of a compact and sufficient set of evidence at the physical level and helping to reduce the context length and inference latency of generative models.

[0084] This embodiment also provides a non-transitory computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the evidence set construction method in the aforementioned retrieval enhancement generation. The non-transitory computer-readable storage medium refers to a tangible medium capable of persistently storing data and excluding transient propagation signals. For example, the storage medium can be a read-only memory, random access memory, disk, optical disk, USB flash drive, portable hard drive, or memory card. When the computer program stored in this medium is loaded into the processor of an electronic device and executed, it can control the electronic device to complete the various steps of the aforementioned method. This implementation, existing in the form of a program product, allows the technical solution to be distributed and deployed independently of a specific hardware platform, facilitating the reproduction of the evidence selection and stop control mechanism based on set state driving in different computing environments, thereby achieving the technical effects of reducing redundant interference, improving response quality, and enhancing traceability.

[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0086] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for constructing an evidence set in retrieval enhancement generation, characterized in that, include: Obtain the user's question and a set of candidate evidence, and initialize the currently selected evidence set to empty; Based on the relationship between the remaining candidate evidence and the currently selected evidence set, the set-related evaluation features of the remaining candidate evidence are determined; wherein, the remaining candidate evidence refers to the evidence in the candidate evidence set other than the currently selected evidence set, and the set-related evaluation features include non-redundancy and new contribution. The non-redundancy characterizes the degree of semantic repetition of the candidate evidence relative to the currently selected evidence set, and the new contribution characterizes the utility increment of the candidate evidence in answering the user question after being added to the currently selected evidence set. Based on the set of currently selected evidence, the remaining candidate evidence, and the set-related evaluation features, a set state is constructed, and a target action is determined in a unified action space; wherein, the unified action space includes candidate selection actions for selecting the remaining candidate evidence and a stop action; In response to the target action being a candidate action, the target candidate evidence indicated by the target action is added to the current selected evidence set. Based on the updated current selected evidence set, the set-related evaluation features of the remaining candidate evidence are re-determined. Then, the process of determining the target action in the unified action space based on the set state constructed based on the current selected evidence set, the remaining candidate evidence, and the set-related evaluation features is returned. In response to the target action being a stop action, the currently selected evidence set is output as the basis for generating retrieval enhancements.

2. The method according to claim 1, characterized in that, The determination of the target action in the unified action space based on the set of currently selected evidence, the remaining candidate evidence, and the set-related evaluation features includes: The set state is processed by a decision model to obtain the predicted scores of each candidate action and the stop action in the unified action space. The target action is determined from the unified action space based on the predicted score; The prediction score of the stop action represents the sufficiency of the currently selected evidence set in answering the user's question.

3. The method according to claim 1, characterized in that, The determination of set-related evaluation features of the remaining candidate evidence based on the relationship between the remaining candidate evidence and the currently selected evidence set includes: For any of the remaining candidate evidence Based on the candidate evidence The candidate semantic vectors and the currently selected evidence set The similarity between the semantic vectors of the selected evidence and the selected evidence in the dataset is used to determine the candidate evidence. The non-redundancy; wherein, when the currently selected evidence set When empty, the non-redundancy is a preset maximum value; when the currently selected evidence set is empty... When the value is not empty, the non-redundancy degree is negatively correlated with the maximum value of the similarity degree; The difference between the first sufficiency estimate and the second sufficiency estimate is determined as the candidate evidence. The newly added contribution; wherein, the first sufficiency estimate is This indicates that the candidate evidence will be... Add to the currently selected evidence set Then, regarding the aforementioned user issues The sufficiency estimate, the second sufficiency estimate is , indicating the currently selected set of evidence. Regarding the aforementioned user issues The sufficiency estimate.

4. The method according to claim 1, characterized in that, The set state includes at least the following information elements: The user issue; The currently selected set of evidence; The remaining candidate evidence; The set of remaining candidate evidence is associated with relevant evaluation features.

5. The method according to claim 1, characterized in that, The acquisition of user questions and candidate evidence sets includes: Receive the initial candidate set output by the external retrieval system; The initial candidate set is mapped to standardized records containing candidate identifiers and candidate content to obtain the candidate evidence set; The mapping process does not modify the internal recall or sorting parameters of the external retrieval system.

6. The method according to claim 5, characterized in that, The initial candidate set received from the external retrieval system includes: Based on the user question, a main search branch and at least one rewritten search branch are constructed. Search operations are performed on each of the search branches to obtain a candidate list for each branch. The candidate lists of each branch are deduplicated and merged according to the candidate identifiers to obtain the initial candidate set; The standardized record also includes at least one of the following: source branch set, cross-branch hit count, intra-branch sorting information, and retrieval score.

7. The method according to claim 1, characterized in that, The method further includes: The evidence in the currently selected evidence set is concatenated according to the order in which they were added to generate context information; The user question and the context information are input into the generation model to obtain the answer to the user question.

8. The method according to claim 1, characterized in that, The method is executed by a decision model, which is trained in the following manner: Obtain the state action trajectory corresponding to the training sample set; wherein, the state action trajectory includes the sample set state and the action label output by the teacher model, the action label includes a sample candidate selection label or a sample stop label, the sample candidate selection label indicates the sample evidence that should be added to the sample selected evidence set, and the sample stop label indicates the termination of the selection process; The student model is trained under supervision based on the state-action trajectory, and the trained student model is used as the decision model. The sample set status includes the sample question, the selected evidence set, the remaining candidate evidence, and the relevant evaluation features of the sample set.

9. The method according to claim 2, characterized in that, The determination of the new contribution in the set-related evaluation features of the remaining candidate evidence includes: For any candidate evidence among the remaining candidate evidence, construct a set of hypothetical evidence after adding the candidate evidence to the currently selected evidence set; Based on the user question, the set of hypothetical evidence, the remaining candidate evidence other than the candidate evidence, and the non-redundancy of the other candidate evidence relative to the set of hypothetical evidence, construct the state of the hypothesis set; The decision model processes the state of the set of assumptions to obtain a first prediction score for the stopping action under the state of the set of assumptions. The decision model is used to process the set state to obtain a second prediction score for the stopping action under the set state; The difference between the first predicted score and the second predicted score is determined as the new contribution of the candidate evidence.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 9.