Retrieval Enhanced Reasoning Model Training Method and Device

Abstract

This disclosure provides a method and apparatus for training a retrieval-enhanced reasoning model, applicable to the field of artificial intelligence. The method includes: acquiring a training sample set containing the question to be reasoned and labeled answers; constructing a structured reasoning template including a plan, search, information, reflection, and answer labels; a strategy model generating an initial plan based on the template to determine the initial evidence acquisition target; generating a search query based on the current reasoning trajectory and sending it to an external retrieval environment, writing the search results as environmental observations into the trajectory, generating reflection content to identify missing or biased evidence states, and dynamically updating the evidence acquisition target accordingly; repeating the above process until a stopping condition is met to generate a final answer; constructing a hierarchical verifiable reward based on the final answer, reflection content, and reasoning trajectory, and training the model using a reinforcement learning algorithm. This disclosure achieves proactive multi-step evidence acquisition, improving the accuracy and stability of complex reasoning tasks.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically to a method and apparatus for training a retrieval-enhanced reasoning model. Background Technology

[0002] In recent years, with the development of artificial intelligence technology, deep learning-based natural language processing (NLP) technology has been widely applied in various scenarios. Large language models, as one of the core technologies in the field of NLP, have demonstrated excellent performance in tasks such as machine translation, text generation, information retrieval, question answering systems, and sentiment analysis. Trained on large-scale corpora, large language models can understand and generate complex patterns in natural language and have achieved significant results in practical applications.

[0003] However, the applicant found that in scenarios such as open-domain question answering, knowledge-intensive reasoning, and multi-hop problem solving, large language models are easily limited by factors such as the static nature of knowledge, insufficient timeliness, and incomplete coverage of long-tail facts due to their reliance on parameterized memory. This leads to problems such as insufficient factual support, interrupted reasoning chains, or generated content deviating from the facts when dealing with complex reasoning tasks.

[0004] To alleviate the aforementioned limitations, retrieval-enhanced generation and retrieval-enhanced reasoning techniques have emerged. These techniques introduce external knowledge acquisition mechanisms during the reasoning process, enabling the model to solve problems by incorporating external evidence. However, existing retrieval-enhanced reasoning techniques still suffer from the following shortcomings in multi-step tasks:

[0005] (i) Lack of global planning in retrieval behavior. Existing methods mainly adopt a passive triggering retrieval strategy, that is, the retrieval is usually triggered by local uncertainty or immediate missing information in the current generation step. Although this approach can supplement immediate facts to a certain extent, due to the lack of a global intermediate goal maintenance mechanism that organizes the overall problem-solving chain, after the model forms incorrect entity assumptions or incorrect intermediate conclusions in the early stage, subsequent retrieval is prone to continue to unfold around the incorrect local assumptions, resulting in fragmented evidence accumulation, deviation of retrieval direction, and mismatch of the reasoning chain.

[0006] (ii) Lack of dynamic target evaluation and correction capabilities. After acquiring new evidence, existing models struggle to proactively assess the current state of evidence and adjust their next search direction accordingly. Search behavior is often fragmented, making it difficult to form a complete chain of evidence, and the model cannot identify missing evidence or reasoning biases and adjust its search strategy in a timely manner.

[0007] (iii) Sparse training signals and insufficient process supervision. The existing training process mainly relies on the outcome reward of whether the final answer is correct, and lacks fine-grained process supervision of intermediate links such as plan generation, retrieval execution, and reflection and correction. Because the reward signals are extremely sparse in multi-step reasoning, the model has difficulty in effectively completing credit allocation in long-chain reasoning, resulting in low training efficiency and difficulty in model convergence.

[0008] (iv) Difficulty in constructing a structured action space. Effectively establishing a unified structured action space within the model and ensuring that multi-round retrievals do not lead to severe contextual redundancy is also a technical challenge. Some studies have attempted to fine-tune retrieval behavior as part of the action space, but still face problems such as fragmented retrieval behavior and incomplete evidence chains.

[0009] In recent years, reinforcement learning-based language model alignment techniques have offered a new approach to solving the aforementioned problems. These techniques can jointly optimize the reasoning strategies of large language models without relying on intensive manual annotation of trajectories, while simultaneously enhancing the model's dynamic knowledge acquisition capabilities in real-world environments by introducing external environmental feedback. However, existing reinforcement learning-based retrieval-enhanced reasoning methods still primarily rely on outcome-based rewards, lacking process supervision of intermediate behaviors such as planning, retrieval, and reflection, making it difficult to achieve proactive, multi-step evidence acquisition-based reasoning strategy optimization.

[0010] Therefore, how to enable the model to actively organize multi-step retrieval and reasoning around the overall evidence chain by introducing structured control mechanisms and fine-grained process rewards, and improve the retrieval organization ability and reasoning stability in solving complex problems, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0011] (a) Technical problems to be solved

[0012] To address at least one of the aforementioned problems, this disclosure provides a method and apparatus for training a retrieval-enhanced reasoning model. By introducing a structured control mechanism of "plan-search-reflection," the model generates a global evidence plan before retrieval and dynamically evaluates the evidence status and actively corrects the retrieval target based on the retrieval results. This transforms passive, locally triggered retrieval into an active, multi-step evidence acquisition process that evolves around the evidence chain. Simultaneously, a hierarchical verifiable reward is constructed to provide fine-grained supervision for intermediate stages such as planning, searching, reflection, and trajectory formatting, alleviating the reward sparsity problem. Furthermore, environmental observation terms are masked during reinforcement learning updates, and a reference model is introduced to apply regularization constraints, suppressing policy drift. This improves the training stability of the retrieval-enhanced reasoning model and the accuracy of answers for complex reasoning tasks.

[0013] (II) Technical Solution

[0014] To address the aforementioned technical problems, embodiments of this disclosure propose a method and apparatus for training a retrieval-enhanced reasoning model.

[0015] According to a first aspect of this disclosure, a method for training a retrieval-enhanced reasoning model is provided, comprising: acquiring a training sample set, the training sample set including a question to be reasoned and labeled answers corresponding to the question to be reasoned; constructing a structured retrieval-enhanced reasoning template, the structured retrieval-enhanced reasoning template including at least plan labels, search labels, information labels, reflection labels, and answer labels; inputting the question to be reasoned into a strategy model to be trained, causing the strategy model to generate an initial plan based on the structured retrieval-enhanced reasoning template, wherein the initial plan is used to determine the initial evidence acquisition target for the question to be reasoned; generating a search query by the strategy model based on the current reasoning trajectory, and sending the search query to an external retrieval environment to obtain retrieval results corresponding to the search query; and writing the retrieval results as environmental observations into the current reasoning. The inference trajectory is generated by the policy model based on the inference trajectory after the retrieval results are written. The inference content is used to identify uncertain information, missing information, or inference bias in the current evidence state, and to maintain, correct, or redirect the current evidence acquisition target accordingly. When the inference content indicates that the retrieval needs to continue, the search query generation, retrieval result writing, and inference content generation are repeated until a preset stopping condition is met, resulting in a complete inference trajectory. Based on the complete inference trajectory, the policy model generates the final answer. Based on the final answer, inference content, and complete inference trajectory, a hierarchical verifiable reward is constructed, and the hierarchical verifiable reward is used as the reward signal for the reinforcement learning algorithm. According to the hierarchical verifiable reward, the policy model is trained and updated using the reinforcement learning algorithm to obtain the target retrieval enhanced inference model.

[0016] In some exemplary embodiments, in the structured retrieval enhanced reasoning template: the plan tag is used to constrain the strategy model to output a high-level evidence acquisition strategy; the search tag is used to constrain the strategy model to generate query statements oriented towards the external retrieval environment; the information tag is used to carry the environmental observations returned by the external retrieval environment, and the content corresponding to the information tag is the observation information returned by the external retrieval environment, which does not belong to the action space generated autonomously by the strategy model; the reflection tag is used to constrain the strategy model to evaluate the current evidence state and update the current evidence acquisition target; and the answer tag is used to constrain the strategy model to output the final answer based on accumulated evidence.

[0017] In some exemplary embodiments, the current inference trajectory includes at least: the problem to be inferred, the initial plan, the generated historical search queries, the search results corresponding to each historical search query, the generated historical reflection content, and the generated intermediate inference content; the strategy model maintains the current evidence acquisition target based on the current inference trajectory, and dynamically updates the current evidence acquisition target based on the newly added environmental observations.

[0018] In some exemplary embodiments, the tiered verifiable reward includes at least an exact match reward, a partial result reward, a reflective perception process reward, and a format reward; wherein, the exact match reward is used to provide the highest level reward when the final answer matches the labeled answer exactly; the partial result reward is used to provide a reward based on the degree of word-level matching when the final answer partially overlaps with the labeled answer; the reflective perception process reward is used to provide a process reward when the final answer does not match the labeled answer exactly but the reflective content contains the labeled answer; and the format reward is used to provide a reward for reasoning trajectories that meet the preset structured label specifications.

[0019] In some exemplary embodiments, generating an initial plan includes: generating a high-level reasoning strategy based on the problem to be reasoned about, which describes a multi-step solution path and the order of evidence acquisition; wherein the high-level reasoning strategy is used to determine an initial chain of evidence before the first retrieval, so that subsequent multiple rounds of retrieval are organized around the initial chain of evidence, rather than triggering retrieval based solely on local uncertainty.

[0020] In some exemplary embodiments, generating reflection content includes: determining whether the currently acquired evidence can support the output of the final answer; identifying uncertain information, missing evidence, or erroneous inferences in the current reasoning process; and, in the case of insufficient current evidence or a deviation in the current search direction, determining the evidence slots to be filled and updating the evidence acquisition targets corresponding to subsequent searches based on the evidence slots to be filled.

[0021] In some exemplary embodiments, the preset stopping conditions include at least one of the following: the reflected content characterizes the current evidence as sufficient to support the output of the final answer; no new valid evidence is obtained in two consecutive rounds of retrieval; or the preset maximum number of retrieval rounds is reached.

[0022] In some exemplary embodiments, training and updating the policy model using a reinforcement learning algorithm based on stratified verifiable rewards includes: sampling and generating multiple candidate inference trajectories for the same inference problem; calculating the stratified verifiable reward corresponding to each candidate inference trajectories; determining the intra-group relative advantage value based on the stratified verifiable reward corresponding to each candidate inference trajectories; and updating the parameters of the policy model based on the intra-group relative advantage value.

[0023] In some exemplary embodiments, updating the policy model parameters based on the intra-group relative advantage value includes: masking the observation terms corresponding to the environmental observations returned by the external retrieval environment so that the loss calculation is focused on the action terms generated by the policy model; and introducing a reference policy model during training to impose regularization constraints on the policy model, thereby suppressing policy drift during training.

[0024] According to a second aspect of this disclosure, a training apparatus for a retrieval-enhanced reasoning model is provided, comprising: a sample acquisition module for acquiring a training sample set, the training sample set including a question to be reasoned and labeled answers corresponding to the question to be reasoned; a template construction module for constructing a structured retrieval-enhanced reasoning template, the structured retrieval-enhanced reasoning template including at least plan labels, search labels, information labels, reflection labels, and answer labels; a plan generation module for inputting the question to be reasoned into a strategy model to be trained, enabling the strategy model to generate an initial plan based on the structured retrieval-enhanced reasoning template, wherein the initial plan is used to determine the initial evidence acquisition target for the question to be reasoned; a retrieval interaction module for generating a search query by the strategy model based on the current reasoning trajectory, and sending the search query to an external retrieval environment to obtain retrieval results corresponding to the search query; and a trajectory update module for writing the retrieval results as environmental observations. The system is divided into several modules: The first module is the Strategies Module ...

[0025] (III) Beneficial Effects

[0026] As can be seen from the above technical solutions, the retrieval enhancement reasoning model training method and apparatus provided in this disclosure have at least the following beneficial effects:

[0027] (1) By introducing a “plan-search-reflect” structured control mechanism, the model can generate a global evidence acquisition plan before the first retrieval and dynamically evaluate the evidence status, identify missing information or reasoning bias based on the external retrieval results in multiple rounds of interaction, thereby actively correcting the subsequent retrieval target. This transforms the traditional passive local trigger retrieval into an active multi-step evidence acquisition process that evolves around the overall evidence chain, significantly improving the retrieval organization ability and evidence chain integrity in complex reasoning tasks.

[0028] (2) By constructing a hierarchical verifiable reward, the supervision of the final answer is combined with the supervision of process behavior, providing fine-grained reward signals for intermediate links such as plan generation, search query, reflection update and trajectory format, which effectively alleviates the problem of sparse reward signals in multi-step reasoning and improves the credit allocation efficiency and convergence stability in the model training process.

[0029] (3) By masking the environmental observation lexical units in the reinforcement learning update, the gradient update of the model parameters is concentrated on the autonomously generated reasoning actions, avoiding the model from misfitting the distribution of the external knowledge base, thereby preventing the language model from degrading; at the same time, the reference policy model is introduced to apply regular constraints to suppress policy drift, ensuring that the model maintains general language understanding and generation capabilities while improving reasoning ability.

[0030] (4) The above mechanisms work together to enable the trained retrieval-enhanced reasoning model to show higher answer accuracy and reasoning stability in complex multi-hop question answering and knowledge-intensive reasoning tasks, and has good practical value and promotion prospects. Attached Figure Description

[0031] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0032] Figure 1 A flowchart illustrating a retrieval-enhanced reasoning model training method for proactive multi-step evidence acquisition according to an embodiment of the present disclosure is shown schematically.

[0033] Figure 2 The illustration schematically shows a logical diagram of an active multi-step evidence acquisition loop process according to an embodiment of the present disclosure;

[0034] Figure 3 This illustration schematically shows the interaction between a strategy model and an external retrieval environment according to an embodiment of the present disclosure;

[0035] Figure 4 This schematically illustrates a construction principle diagram of a tiered verifiable reward according to an embodiment of the present disclosure;

[0036] Figure 5 This illustration schematically depicts a reinforcement learning training update based on intra-group relative advantage according to an embodiment of the present disclosure;

[0037] Figure 6 This schematic diagram illustrates the structural block diagram of a retrieval-enhanced reasoning model training apparatus for proactive multi-step evidence acquisition according to an embodiment of the present disclosure;

[0038] Figure 7A block diagram of an electronic device for a retrieval-enhanced reasoning model training method for proactive multi-step evidence acquisition according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0039] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0041] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0042] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0043] Figure 1 A flowchart illustrating a retrieval-enhanced reasoning model training method for proactive multi-step evidence acquisition according to an embodiment of the present disclosure is shown.

[0044] like Figure 1 As shown, the retrieval enhancement reasoning model training method for proactive multi-step evidence acquisition according to the embodiments of this disclosure includes steps S110 to S190.

[0045] In step S110, a training sample set is obtained, which includes the question to be reasoned and the labeled answer corresponding to the question to be reasoned.

[0046] In the embodiments of this disclosure, the questions to be reasoned include knowledge-intensive multi-hop question-and-answer questions, such as cross-document entity links and implicit relation derivations, which cannot be answered solely by the model's internal parameterized memory. Abnormal character filtering and format normalization are performed on the question text, and entity aliasing and lexicalization are performed on the labeled answers to provide high-quality state input for subsequent reinforcement learning training.

[0047] In multi-hop complex reasoning tasks, model training requires high-quality question-answer pairs as supervision signals. This step collects a sample set containing the question to be reasoned and its corresponding standard labeled answer, providing environmental feedback for subsequent reinforcement learning. Simultaneously, filtering for abnormal characters and normalizing the format of the question text, and unifying entity aliases and lexicalizing the labeled answers, reduces data noise and ensures the standardization and consistency of state inputs. This standardized preprocessing improves the quality of training data, reduces training fluctuations caused by inconsistent data formats or ambiguous entity aliases, and lays the foundation for the model to stably learn reasoning strategies.

[0048] In step S120, a structured retrieval enhanced reasoning template is constructed, which includes at least plan tags, search tags, information tags, reflection tags, and answer tags.

[0049] In this embodiment, the plan tag (plan: ) is used to constrain the policy model output of the high-level evidence acquisition strategy; search tags (search: ) is used to constrain the strategy model to generate query statements for the external retrieval environment; information tags (information: The information tag is used to carry environmental observations returned by the external retrieval environment, and the content corresponding to the information tag is the observation information returned by the external retrieval environment, which does not belong to the action space generated autonomously by the policy model; the reflection tag (reflect: ) is used to constrain the policy model to evaluate the current evidence state and update the current evidence acquisition goal; answer tags (answer: This is used to constrain the policy model output to the final answer based on accumulated evidence.

[0050] Based on the template, the complete inference trajectory generated by the policy model can be formally represented as:

[0051]

[0052] in, Indicates the initial plan, Indicates the first Search queries generated in rounds, This indicates the observation information returned by the external retrieval environment. Indicates the first The results of the reflection generated by the cycle This indicates the final predicted answer. This refers to the actual number of interaction rounds in a multi-round retrieval process.

[0053] To transform multi-round retrieval behavior into a Markov decision process that can be learned by the policy model, a clear action space and observation space need to be defined. This step forces the model to output inference trajectories according to a fixed paradigm by designing an XML-style template containing labels for plan, search, information, reflection, and answer. Specifically, the plan label corresponds to the global policy, the search label corresponds to the action, the information label carries environmental feedback, the reflection label is used for state evaluation, and the answer label outputs the final result. By strictly separating external observations from internal actions, the learning boundaries of the model are clearly defined.

[0054] The structured templates give the reasoning trajectory a uniform format, which facilitates subsequent hierarchical reward calculation and training signal extraction. At the same time, forcing the model to follow the sequence of "plan-action-observation-reflection-answer" helps the model form stable reasoning habits and reduces invalid output.

[0055] In step S130, the problem to be reasoned is input into the policy model to be trained, and the policy model generates an initial plan based on the structured retrieval enhanced reasoning template. The initial plan is used to determine the initial evidence acquisition target for the problem to be reasoned.

[0056] In embodiments of this disclosure, generating an initial plan includes: generating a high-level reasoning strategy based on the problem to be reasoned about, which describes the multi-step solution path and the order of evidence acquisition; wherein the high-level reasoning strategy is used to determine an initial chain of evidence before the first retrieval, so that subsequent multiple rounds of retrieval are organized around the initial chain of evidence, rather than triggering retrieval solely based on local uncertainty.

[0057] Traditional retrieval augmentation methods often passively trigger retrieval when encountering uncertainty during the generation process, lacking global guidance. This step forces the model to output a high-level evidence acquisition strategy before the first retrieval, decomposing the complex problem into ordered sub-objectives (e.g., entity recognition → relation extraction → numerical computation). This "plan before you act" mechanism simulates human reasoning habits, enabling the model to establish a priori cognitive scaffold.

[0058] The initial plan provides a clear direction for subsequent rounds of retrieval, avoiding semantic drift caused by misleading local keywords; at the same time, the decomposition of the plan helps to allocate the credit of long-range reasoning to each sub-objective, alleviating the reward sparsity problem.

[0059] For example, when the problem to be reasoned about is "How many plagues occurred in the place where the artist of the painting LaBella died?", traditional retrieval-enhanced reasoning methods often directly capture the local keyword "number of plagues" for a greedy search, causing the search results to deviate from the context of the painting itself. In this embodiment, however, the strategy model first outputs an explicit logical chain within the plan tag: Step 1, find the identity of the creator of the painting LaBella; Step 2, retrieve the creator's biography and confirm the geographical location of their death; Step 3, combine the geographical location information to count the specific number of plagues that occurred in that area throughout history. This planned mechanism allows subsequent rounds of retrieval to proceed in an organized manner around the overall solution chain, effectively avoiding the semantic drift problem caused by excessively high similarity of local entities in the early reasoning stage.

[0060] In step S140, based on the current inference trajectory, a search query is generated by the strategy model and sent to an external retrieval environment to obtain the retrieval results corresponding to the search query.

[0061] In the embodiments of this disclosure, the current reasoning trajectory includes at least: the problem to be reasoned, the initial plan, the generated historical search queries, the search results corresponding to each historical search query, the generated historical reflection content, and the generated intermediate reasoning content; the strategy model maintains the current evidence acquisition target according to the current reasoning trajectory, and dynamically updates the current evidence acquisition target according to the newly added environmental observations.

[0062] The strategy model generates targeted search queries based on the currently accumulated reasoning trajectory (including questions, plans, historical search results, and reflections). These queries are sent to external retrieval environments (such as the Wikipedia API or the BM25 search engine), which return the most relevant text snippets as observations. This process can be viewed as the model actively seeking information from external knowledge sources, and the quality of these queries directly impacts the accuracy of subsequent reasoning.

[0063] By dynamically maintaining the inference trajectory, the model can adjust its query direction based on the acquired evidence, achieving "proactive" retrieval rather than passively relying on local uncertainties. Environmental feedback provides the model with objective facts, reducing the illusions caused by parametric memory.

[0064] For example, as reasoning progresses, the policy model generates targeted search queries based on the "current reasoning trajectory" maintained in its context window (i.e., containing all historical sequences from operation S110 to the previous round of reflection). The query was placed within a search tag and sent to an external retrieval environment. (For example, a dense vector retrieval library or the BM25 sparse retrieval engine mounted with an offline Wikipedia corpus). After receiving the query, the external retrieval environment performs similarity matching and recalls the Top-K highly relevant text fragments, which are then used as environmental observations. Return. This interaction process can be mathematically abstracted as a deterministic environment state transition function: .

[0065] In step S150, the retrieval results are written into the current inference trajectory as environmental observations. Based on the inference trajectory after the retrieval results are written, the strategy model generates reflection content. The reflection content is used to identify uncertain information, missing information, or inference bias in the current evidence state, and to maintain, correct, or redirect the current evidence acquisition target accordingly.

[0066] In the embodiments of this disclosure, generating reflection content includes: determining whether the currently acquired evidence can support the output of the final answer; identifying uncertain information, missing evidence, or erroneous inferences in the current reasoning process; and, in the case of insufficient current evidence or deviation in the current search direction, determining the evidence slots to be filled and updating the evidence acquisition targets corresponding to the subsequent searches based on the evidence slots to be filled.

[0067] According to embodiments of this disclosure, generating reflection content may include, for example,: logical consistency assessment, determining whether all currently acquired external evidence is sufficient to form a logical loop and support the output of a precise final answer; conflict and missing information identification, identifying uncertain information, conflicting evidence, or erroneous inferences in the model's initial plan that exist in the current reasoning trajectory; and target redirection, in cases where it is determined that the current evidence is insufficient or the search direction has shifted, clearly indicating the key evidence slots to be filled, and redirecting the direct target of the next round of search based on the slots.

[0068] The reflective mechanism is the core of proactive evidence acquisition. It enables the model to promptly identify deviations in the reasoning direction and avoid continuously searching based on erroneous assumptions. By clearly identifying missing evidence slots, the model can generate more accurate subsequent queries, improving retrieval efficiency.

[0069] Continuing with the scenario example in step S140: Assume the first round of search ( The observation information returned by "LaBellaauthor" The text indicates that the author is Titian, but makes no mention of the plague. At this point, the model generates a reflection within the `reflect` tag. The current evidence confirms the author as Titian, but information on Titian's death site is missing. The search query needs to be revised to confirm Titian's death site. This dynamic redirection based on status feedback is the underlying principle behind this disclosure's "proactive" evidence acquisition mechanism.

[0070] In step S160, when the reflection content indicates that retrieval needs to continue, the search query generation, retrieval result writing, and reflection content generation are repeatedly executed until the preset stopping condition is met, and a complete reasoning trajectory is obtained.

[0071] In the embodiments of this disclosure, the preset stopping conditions include at least one of the following: the reflected content characterizes that the current evidence is sufficient to support the output of the final answer; no new valid evidence is obtained in two consecutive rounds of retrieval; or the preset maximum number of retrieval rounds is reached.

[0072] After safely exiting the "search-observation-reflection" loop by meeting the above stopping conditions, the policy model will synthesize all valid evidence in the context and output the final structured solution in the answer label.

[0073] When the reflection indicates that further retrieval is needed, the model repeats steps S140 and S150, forming a closed loop of "search-observation-reflection-redirection". The loop terminates when the reflection indicates sufficient evidence, no new information is found in two consecutive rounds, or the maximum number of rounds is reached. This mechanism balances inference depth and efficiency. Through adaptive stopping conditions, the model can terminate the retrieval in a timely manner when evidence is sufficient, avoiding unnecessary computational overhead; at the same time, the maximum number of rounds constraint prevents infinite loops. This loop allows the model to gradually improve the chain of evidence until it can support the final answer.

[0074] In step S170, the final answer is generated by the policy model based on the complete reasoning trajectory.

[0075] After the loop terminates, the model generates a final answer based on the complete reasoning trajectory (including all plans, retrieval results, and reflections). The answer must be consistent with the accumulated evidence and is typically required to be in a standardized format (e.g., numbers, entity names, etc.). Because the final answer is built on a chain of evidence validated through reflection, its accuracy is far higher than that of direct parametric memory or single retrieval results, effectively reducing the illusion phenomenon.

[0076] In step S180, a hierarchical verifiable reward is constructed based on the final answer, the reflection content, and the complete reasoning trajectory, and the hierarchical verifiable reward is used as the reward signal for the reinforcement learning algorithm.

[0077] In the embodiments of this disclosure, to address the reward sparsity problem in long-range inference, this step constructs a multi-level reward function. The hierarchical verifiable reward includes at least an exact match reward, a partial result reward, a reflective perception process reward, and a format reward. Specifically, the exact match reward provides the highest-level reward when the final answer precisely matches the labeled answer; the partial result reward provides a reward based on the word-level matching degree when the final answer partially overlaps with the labeled answer; the reflective perception process reward provides a process reward when the final answer does not precisely match the labeled answer but the reflective content contains the labeled answer; and the format reward provides a reward for inference trajectories that meet the preset structured label specifications. The hierarchical rewards provide fine-grained credit allocation signals, enabling the model to learn from partially correct trajectories and accelerate convergence; the reflective perception reward is particularly important, encouraging the model to correctly identify key information in reflection even if it cannot fully answer.

[0078] In the embodiments of this disclosure, to overcome the problem of sparse reward signals in reinforcement learning during long-chain multi-hop inference (i.e., rewarding solely based on the correctness of the final answer leads to the failure of credit allocation), this disclosure extracts the complete inference trajectory generated by the policy model and constructs a hierarchical verifiable reward that deeply integrates end-result supervision with process behavior guidance. This reward mechanism not only evaluates the accuracy of the final answer but also penetrates down to evaluate whether key facts were captured during the reflection process. Subsequently, a group relative policy optimization algorithm is used to sample multiple trajectories for the same problem to calculate the relative advantage value. During gradient calculation, the external environment observation lexical in the information label is masked to ensure that the backpropagation of the loss function only applies to the plan, search, reflection, and answer lexical autonomously generated by the policy model, thereby achieving efficient and stable alignment of the inference strategy of the large language model.

[0079] In step S190, the policy model is trained and updated using a reinforcement learning algorithm based on the hierarchical verifiable reward to obtain the target retrieval enhanced reasoning model.

[0080] In the embodiments of this disclosure, step S190 may specifically include: sampling and generating multiple candidate reasoning trajectories for the same reasoning problem; calculating the stratified verifiable reward corresponding to each candidate reasoning trajectory; determining the relative advantage value within the group based on the stratified verifiable reward corresponding to each candidate reasoning trajectory; and updating the execution parameters of the policy model based on the relative advantage value within the group.

[0081] The parameter updates for the policy model based on the intra-group relative advantage value include: masking the observation terms corresponding to the environmental observations returned by the external retrieval environment so that the loss calculation is focused on the action terms generated by the policy model; and introducing a reference policy model during training to impose regularization constraints on the policy model, thereby suppressing policy drift during training. Masking avoids the model misfitting the external knowledge base and maintains the generative ability of the language model; KL constraints ensure training stability and prevent model degradation; and intra-group relative advantage eliminates the need for value network training, improving computational efficiency.

[0082] Figure 2 The illustration schematically shows a logical diagram of an active multi-step evidence acquisition loop process according to an embodiment of the present disclosure.

[0083] like Figure 2 As shown, the cyclical process specifically includes a planning phase, a search and interaction phase, a reflection and update phase, and an answer generation phase, and includes a judgment node that determines the trajectory.

[0084] like Figure 2 As shown, according to the embodiments of this disclosure, the multi-step evidence acquisition process is strictly divided into four core stages with a logical sequence: planning stage, search and interaction stage, reflection and update stage, and answer generation stage, and includes a judgment node that determines the trajectory.

[0085] First, we enter Phase 1: the planning phase. Upon receiving the question to be reasoned about, the strategy model does not immediately extract local keywords to trigger external searches, but instead forces itself into a global planning mode. Within the "plan" label, the strategy model outputs a high-level evidence acquisition strategy, determining the initial evidence chain for multi-step solutions. The core purpose of this planning phase is to establish a "prior cognitive scaffold" for the model. Complex problems are deconstructed into multiple ordered sub-objectives (e.g., attribute extraction of entity A → association comparison of entity B → final numerical calculation). This mechanism ensures that the model establishes a global retrieval objective in the initial stage, fundamentally overcoming the semantic drift and early retrieval direction deviation problems caused by local keyword-triggered searches in traditional methods. This allows subsequent rounds of retrieval to unfold systematically around a complete evidence chain, significantly improving the consistency of the retrieval direction and the completeness of the evidence chain.

[0086] Subsequently, the second stage, the search and information stage, begins. After establishing the initial chain of evidence, the model outputs a precise query statement within the `search` tag, based on the currently determined direct objective. This query statement is then detached from the model and sent to the external retrieval environment. The external retrieval environment responds to the query statement, performs document retrieval, and returns the corresponding text fragments. This externally returned data is strictly confined within the `information` tag and written into the current inference trajectory. By treating `search` and `information` as an "action-observation" pair, the large language model can perceive real feedback from the external world like an intelligent agent. Its beneficial effects are: by strictly separating external observation from internal actions, it provides objective factual anchors for subsequent logical deduction, avoiding the model's illusions arising solely from parametric memory; simultaneously, the dynamically maintained inference trajectory allows the model to adjust its query direction based on the acquired evidence, realizing a shift from passive local triggering to proactive information acquisition.

[0087] Next, we move to the third stage: the reflection stage. This is the most crucial step in achieving "proactive" information gathering. After each new environmental observation, the model is forced to enter the "reflect" tab for logical introspection, comparing and evaluating all external evidence accumulated in the current "information" tab with the global objective set in the "plan" tab or the sub-objective of the previous retargeting round. The reflection content is not only a summary of known information but also a diagnosis of unknown or missing information. For example, the model might identify that "the current evidence only confirms the existence of entity A, but lacks the interaction time between entity A and entity B," thus proactively pointing out the break in the current evidence chain. Its beneficial effects are: enabling the model to promptly detect deviations in the reasoning direction and gaps in evidence, avoiding continuous searching on erroneous assumptions, and providing precise missing information slots for subsequent retargeting; simultaneously, the reflection mechanism distributes the credit of long-range reasoning to each intermediate step, providing a process supervision basis for subsequent tiered rewards and effectively alleviating the reward sparsity problem.

[0088] Then, the logical judgment node is reached: Is the evidence sufficient to support the answer? Based on the diagnostic results of the reflection phase, the model faces a branching decision:

[0089] Branch 1 (Redirection and Loop): When the content of the reflection indicates that the current evidence is missing, conflicting, or insufficient to support the response ( Figure 2 (In the "No" branch), the model triggers a redirection mechanism, dynamically updating the direct retrieval target for the next round based on the "missing information slots" identified in the reflection phase, and generating a new search statement accordingly, guiding the trajectory along... Figure 2The loop path on the left traces back to the search interaction stage. This closed loop of "search → observation → reflection → target retargeting → search" constitutes the core driving force of this disclosure. Its beneficial effects are: the model will not blindly repeat the previous round of search, but will actively correct the search direction based on the diagnostic results, thereby effectively avoiding error accumulation and waste of search resources, and improving the efficiency and accuracy of multi-round searches; the closed-loop mechanism enables the model to continuously self-correct during exploration and gradually improve the evidence chain.

[0090] Branch Two (Entering the Final Stage): The conclusion is reached if and only if the content of the reflection represents a complete and logically rigorous chain of evidence that is sufficient to logically deduce the final conclusion. Figure 2 (If the "yes" branch is selected), the loop terminates, and the trajectory flows to the answer generation stage. Its beneficial effect is that, through adaptive stopping conditions, the model can terminate the retrieval in a timely manner when the evidence is sufficient, balancing inference depth and computational overhead, and avoiding the resource consumption caused by unlimited retrieval.

[0091] Finally, we move to stage 4: the answer generation stage. After satisfying the aforementioned condition of sufficient evidence, the policy model integrates all valid reasoning threads and factual evidence within the context of the answer label, outputting a structured final answer supported by solid evidence. Its beneficial effects are: because the final answer is built on a complete chain of evidence that has been reflected upon and verified, it eliminates the local guessing (i.e., illusion) that traditional large language models exhibit in knowledge blind spots, significantly improving the accuracy and interpretability of the answer.

[0092] pass Figure 2 The planned-search-reflection cycle process shown in this disclosure effectively weaves the originally fragmented and passive retrieval actions into an exploration network driven by internal cognitive goals in real time. This greatly improves the reliability of the model's solutions when dealing with long-tail knowledge and complex multi-hop logic, and realizes a paradigm shift from passive local triggering to active global evidence acquisition.

[0093] Combination Figure 1 Operation S140 and Figure 2 The PSR cycle in Figure 3 The diagram illustrates the interaction between a strategy model and an external retrieval environment according to an embodiment of the present disclosure.

[0094] like Figure 3 As shown, according to an embodiment of this disclosure, the entire interactive system consists of two decoupled parts that communicate through a structured interface: a strategy model located at the logical core and an external retrieval environment E located on the external environment side.

[0095] First, the internal interaction logic of the strategy model is explained in detail. The strategy model is a parameterized language model to be trained. The core task of the policy model is to maintain and update the current inference trajectory in real time.

[0096] According to embodiments of this disclosure, during the reasoning process, the strategy model stores all generated plans, historical searches, historical reflections, and observational information obtained from the external environment as contextual states in its reasoning trajectory. In each retrieval step... In the process, the strategy model scans the current inference trajectory through its internal attention mechanism to identify the missing evidence points that need to be supplemented. Subsequently, the strategy model outputs search tag pairs that conform to preset specifications within the inference trajectory, and generates targeted search queries within these tag pairs. This search query As a proactively emitted "logic probe," it represents the information demand that the model actively sends to the external world to advance reasoning. Subsequently, the process of sending search queries from the policy model to the external retrieval environment is explained. The search query generated by the policy model... The data is extracted in real time and sent to an external retrieval environment, which can be achieved by calling application programming interfaces or internal retrieval functions. Search query The forms include: dense feature representations for vector retrieval systems; sparse text queries for keyword matching systems (e.g., queries based on the BM25 algorithm); and structured query languages ​​or knowledge graph queries for structured databases. Its advantages include: flexible adaptation to multiple query formats, enabling this method to be compatible with different types of retrieval backends, enhancing the system's versatility and scalability; and structured interfaces ensuring efficient decoupling of the strategy model from the external environment, facilitating system maintenance and upgrades.

[0097] Next, the response and retrieval logic of the external retrieval environment will be explained. The external retrieval environment carries a vast amount of external knowledge beyond the model parameters (such as full-text encyclopedia databases, internal enterprise document libraries, professional knowledge bases, or internet search engines). The external retrieval environment receives search queries. Then, using a pre-set recall algorithm (such as multi-path recall technology), a rapid matching is performed on the massive corpus to retrieve the text fragments, document summaries, or fact entries with the highest semantic relevance to the query, and these raw facts are encapsulated as environmental observations. Its beneficial effects are as follows: by leveraging external knowledge sources, the model can overcome the static and time-sensitive limitations of parametric memory, obtain the latest and most relevant factual evidence, and significantly reduce the risk of generated content deviating from the facts; multi-path recall technology improves the recall rate and accuracy of retrieval, providing a more reliable evidentiary basis for subsequent reflection and answer generation.

[0098] Finally, the process of transmitting and writing environmental observations to the policy model is explained. The external retrieval environment will generate environmental observations. The data is fed back to the policy model, and the system automatically incorporates environmental observations. Fill in the information label for the current inference trajectory. Return environmental observations. This not only enriches the contextual information of the policy model but also provides objective "external truth" feedback. Because the content within the information tags is defined as an unchangeable observed state, the policy model must use this as a basis for logical alignment when generating reflective content or the final answer in the next round. This "sending demand" approach... Obtain the facts The "write state" architecture enables a fundamental shift in the model from simple "probabilistic prediction" to "objective evidence-based reasoning." By forcibly writing environmental observations into unchangeable labels, it prevents the model from arbitrarily altering facts and enhances the transparency and verifiability of the reasoning process. At the same time, this design provides factual anchors for the reflective perception process reward in the hierarchical verifiable reward system, enabling the model to learn from the trajectory of "correct reflection even though the answer is wrong," effectively alleviating the credit allocation problem.

[0099] pass Figure 3 The interactive architecture shown in this disclosure constructs an intelligent reasoning system with closed-loop feedback capabilities. This enables the strategy model to, like a human expert, gradually peel away the fog of confusion and ultimately approach the truth when faced with complex and unknown problems through effective communication with external knowledge sources. This system not only improves the accuracy of the retrieved and enhanced reasoning model's answers but also enhances its robustness and generalization ability in scenarios involving long-tail knowledge and multi-hop logic.

[0100] Figure 4 A schematic diagram illustrating the construction principle of a hierarchical verifiable reward system according to an embodiment of this disclosure is provided. Figure 1 Steps S180 and S190 in the process, Figure 4 This further demonstrates how the present disclosure provides fine-grained supervision signals for the multi-step inference trajectory of the policy model through multi-dimensional verification logic.

[0101] In the embodiments of this disclosure, the core motivation for constructing hierarchical verifiable rewards lies in addressing the technical problem of extremely sparse reward signals in long-range reasoning tasks. Traditional reinforcement learning only provides binary rewards (0 or 1) based on the correctness of the final answer, making it difficult for the model to obtain timely positive incentives for effective actions in intermediate retrieval and reflection steps. To solve this problem, this disclosure divides the rewards into four levels with strictly decreasing priority, achieving a comprehensive evaluation of the reasoning trajectory. Its beneficial effects are: providing the model with progressive learning signals from structural compliance and evidence acquisition effectiveness to answer accuracy, significantly improving credit allocation efficiency and training convergence speed.

[0102] First, let's explain the first level: exact match reward. This applies when the system determines the final answer generated by the strategy model. With the marked answer Achieve exact match (i.e.) When this occurs, the highest-level reward is triggered. The formula for calculating this level of reward could be, for example: .in, The preset base reward value; This is the label penalty coefficient, used to apply negative feedback to duplicate answer blocks or anomalous label distributions in the inference trajectory, thereby guiding the model to generate concise and logically sound outputs. This level aims to reward high-quality trajectories with complete logical chains and absolutely correct final results.

[0103] The second level, partial result rewards, is then explained in detail. This applies when the final answer does not achieve an exact match, but intersects with the labeled answer at the lexical level (i.e., at the lexical level). When the value is greater than zero, a partial reward is triggered. The reward calculation formula for this level can be, for example: .in, This is the preset first weighting coefficient. This level introduces continuous... The score provides a smooth gradient signal for the "close to correct" answer, effectively mitigating training fluctuations caused by discrete rewards, helping the model quickly approach the correct answer in the early stages of training, and accelerating the initial convergence of the policy.

[0104] Next, the third level, rewarding the reflective perception process, will be explained in detail. This level is an innovation in the field of process supervision. When the final answer does not match the labeled answer at all, but the system detects the labeled answer... Correctly included in the reflective content generated by the strategy model During this period, a reward is triggered for the reflective and perceptual process. The reward calculation formula for this level could be, for example: .in, This is the preset second weight coefficient. The technical significance of this level is that it can identify those trajectories that "have retrieved the correct evidence but failed in the final step of summarizing." By giving positive incentives to trajectories with high-value intermediate information, it guides the model to learn how to correctly utilize the retrieved evidence, greatly improving the credit allocation efficiency in multi-step reasoning, and enabling the model to obtain effective learning signals even if the final answer is wrong.

[0105] Finally, the fourth level, the basic format reward, is explained in detail. When none of the above three matching conditions are met, if the inference trajectory τ still strictly follows the structured labeling specifications of plan, search, reflect, and answer, then the basic format reward is triggered. This formatted reward serves as a "safety net reward," designed to constrain the output distribution of the policy model and ensure it doesn't deviate from the pre-defined inference paradigm during reinforcement learning exploration. It guarantees the resolvability of the inference trajectory, providing fundamental guidance for the model's transition from "random generation" to "structured reasoning," while also ensuring the reliable computation of higher-level rewards.

[0106] pass Figure 4 The layered verifiable reward mechanism shown in this disclosure successfully transforms a single final evaluation into a full-dimensional evaluation system covering "structural stability, evidence acquisition effectiveness, and result accuracy," fundamentally improving the training efficiency and generalization ability of the retrieval-enhanced reasoning model in complex environments and realizing refined process supervision of the model's multi-step reasoning behavior.

[0107] Figure 5 This illustration schematically depicts a reinforcement learning training update based on intra-group relative advantage according to an embodiment of the present disclosure. (In conjunction with...) Figure 1 Step S190 and Figure 4 The tiered structure shown verifies the rewards. Figure 5 It further demonstrates how to efficiently and stably update the network parameters of the policy model using gradients after obtaining the reward signal.

[0108] Because the multi-step retrieval-enhanced reasoning process involves discrete natural language generation actions and non-differentiable interactions with the external environment, traditional supervised learning struggles to directly optimize the entire reasoning chain. Therefore, this disclosure employs a group-relative policy optimization algorithm to achieve global policy alignment. For example... Figure 5 As shown, the reinforcement learning update process includes the following four key stages.

[0109] First, the multi-trajectory sampling process within the group is explained in detail. In each training iteration, for the same input problem to be inferred... The current policy model to be trained Perform multiple independent autoregressive samplings to generate A set of candidate reasoning trajectories ,in The preset group size (e.g., 4 to 8) allows the model to explore different planning paths, search keywords, and reflection logics by sampling the same problem in diverse ways. This collects rich action samples, providing a sufficient foundation of strategy diversity for subsequent relative advantage evaluation and preventing the model from getting stuck in local optima.

[0110] Subsequently, a detailed explanation of the tiered reward calculation and advantage value assessment process is provided. (This is followed by a continuation of the previous sentence, which is incomplete and requires further context.) After identifying candidate trajectories, the system calls the hierarchical verifiable reward calculation module to calculate the reward score set corresponding to each trajectory. To reduce variance during reinforcement learning training, this disclosure does not rely on absolute reward values ​​for parameter updates, but instead employs a within-group relative advantage mechanism. Specifically, it calculates... The average experience of each reward value and standard deviation The reward for each trajectory is standardized to obtain the relative advantage value within the group. .like This indicates that the inference strategy for this trajectory is better than the average level of the same group, and the system will increase the probability of generating this trajectory in subsequent updates; if If the result is negative, it indicates that the strategy is performing poorly, and the system will suppress the generation of similar trajectories. This design eliminates the need for complex value network training in traditional reinforcement learning, significantly reducing computational overhead and memory usage.

[0111] Next, the environmental observation lexical masking process is explained in detail. The structured inference trajectory includes internal action lexicals generated by the model itself (such as text within the labels "plan," "search," "reflect," and "answer") and observation lexicals generated by the external environment (such as text returned from the knowledge base within the "information" label). When calculating the cross-entropy loss and gradient backpropagation of the policy model, strict masking is performed on the observation lexicals returned by the external retrieval environment; that is, the weights of these lexicals in the loss function are reset to 0. The underlying logic of this masking mechanism is that external observations are feedback from the objective environment, not decision actions of the policy model; without masking, the model will incorrectly fit the text distribution of the external knowledge base, leading to language model degradation. Its beneficial effects are: ensuring that gradient updates focus on the model-generated plan, search, reflection, and answer lexicals, thereby accurately optimizing the inference strategy while maintaining the general generative capability of the language model.

[0112] Finally, the model parameter update and policy drift constraint processes are explained in detail. After completing the masking process and calculating the relative advantage values ​​within the group, the system constructs the reinforcement learning objective function and performs gradient ascent to update the parameters of the policy model. To prevent "policy drift" in the pursuit of high reward scores (e.g., outputting repetitive shortcut text to cheat for rewards, disrupting natural language fluency, or completely forgetting pre-trained knowledge), this disclosure introduces a reference policy model with frozen weights into the training objective. (Typically, this is the initial pre-trained model). At each generation step, the current policy model is computed. Compared with the reference strategy model The KL divergence between the output probability distributions is incorporated as a penalty term into the total loss function. This regularization constraint effectively limits the step size of a single parameter update, ensuring that the model maintains high-quality general language understanding and generation capabilities while learning proactive multi-step retrieval strategies.

[0113] pass Figure 5 The training and update architecture shown in this disclosure implements a highly stable, low-variance reinforcement learning alignment mechanism that strictly shields environmental noise, enabling large language models to truly master the complex reasoning paradigm of "calling external tools and reflecting like humans".

[0114] In combination with the above Figures 1 to 5 Method and process Figure 6 From the perspective of functional modular design, the logical architecture of this disclosed system during actual deployment is further demonstrated.

[0115] Figure 6 A schematic block diagram of a retrieval-enhanced reasoning model training apparatus for proactive multi-step evidence acquisition according to an embodiment of the present disclosure is shown.

[0116] like Figure 6 As shown, the retrieval-enhanced reasoning model training device 600 for proactive multi-step evidence acquisition in this embodiment includes a sample acquisition module 610, a template construction module 620, a plan generation module 630, a retrieval interaction module 640, a trajectory update module 650, a reflection update module 660, an answer generation module 670, a reward construction module 680, and a model training module 690.

[0117] The sample acquisition module 610 is used to acquire a training sample set, which includes the question to be reasoned and the labeled answers corresponding to the question. In one embodiment, the sample acquisition module 610 can be used to perform the operation S110 described above, which will not be repeated here.

[0118] The template construction module 620 is used to construct a structured retrieval enhanced reasoning template, which includes at least plan tags, search tags, information tags, reflection tags, and answer tags. In one embodiment, the template construction module 620 can be used to perform the operation S120 described above, which will not be repeated here.

[0119] The plan generation module 630 is used to input the problem to be reasoned into the policy model to be trained, so that the policy model generates an initial plan based on the structured retrieval enhanced reasoning template. The initial plan is used to determine the initial evidence acquisition target for the problem to be reasoned. In one embodiment, the plan generation module 630 can be used to perform the operation S130 described above, which will not be repeated here.

[0120] The retrieval interaction module 640 is used to generate a search query from the strategy model based on the current inference trajectory, and send the search query to an external retrieval environment to obtain the search results corresponding to the search query. In one embodiment, the retrieval interaction module 640 can be used to perform the operation S140 described above, which will not be repeated here.

[0121] The trajectory update module 650 is used to write the retrieval results as environmental observations into the current inference trajectory, and based on the inference trajectory after writing the retrieval results, the strategy model generates reflection content. The reflection content is used to identify uncertain information, missing information, or inference biases in the current evidence state, and accordingly maintain, correct, or redirect the current evidence acquisition target. In one embodiment, the trajectory update module 650 can be used to perform the operation S150 described above, which will not be repeated here.

[0122] The reflection update module 660 is used to repeatedly execute the search query generation, search result writing, and reflection content generation when the reflection content indicates that retrieval needs to continue, until a preset stopping condition is met, thus obtaining a complete reasoning trajectory. In one embodiment, the reflection update module 660 can be used to execute the operation S160 described above, which will not be repeated here.

[0123] The answer generation module 670 is used to generate the final answer from the policy model based on the complete reasoning trajectory. In one embodiment, the answer generation module 670 can be used to perform the operation S170 described above, which will not be repeated here.

[0124] The reward construction module 680 is used to construct a hierarchical verifiable reward based on the final answer, reflection content, and complete reasoning trajectory, and to use the hierarchical verifiable reward as the reward signal for the reinforcement learning algorithm. In one embodiment, the reward construction module 680 can be used to perform the operation S180 described above, which will not be repeated here.

[0125] The model training module 690 is used to train and update the policy model using a reinforcement learning algorithm based on hierarchical verifiable rewards, thereby obtaining an enhanced inference model for target retrieval. In one embodiment, the model training module 690 can be used to perform the operation S190 described above, which will not be repeated here.

[0126] According to embodiments of this disclosure, any and multiple modules among the sample acquisition module 610, template construction module 620, plan generation module 630, retrieval interaction module 640, trajectory update module 650, reflection update module 660, answer generation module 670, reward construction module 680, and model training module 690 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the sample acquisition module 610, template construction module 620, plan generation module 630, retrieval interaction module 640, trajectory update module 650, reflection update module 660, answer generation module 670, reward construction module 680, and model training module 690 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the sample acquisition module 610, template construction module 620, plan generation module 630, retrieval interaction module 640, trajectory update module 650, reflection update module 660, answer generation module 670, reward construction module 680, and model training module 690 can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0127] Figure 7 A block diagram of an electronic device for a retrieval-enhanced reasoning model training method for proactive multi-step evidence acquisition according to an embodiment of the present disclosure is shown schematically.

[0128] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0129] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0130] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0131] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0132] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0133] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

Claims

1. A retrieval-augmented inference model training method, characterized in that, include: Obtain a training sample set, which includes the question to be reasoned and the labeled answer corresponding to the question to be reasoned; Construct a structured retrieval enhanced reasoning template, which includes at least plan tags, search tags, information tags, reflection tags, and answer tags; The problem to be reasoned is input into the policy model to be trained, and the policy model generates an initial plan based on the structured retrieval enhanced reasoning template, wherein the initial plan is used to determine the initial evidence acquisition target for the problem to be reasoned. Based on the current inference trajectory, the strategy model generates a search query and sends the search query to an external retrieval environment to obtain the retrieval results corresponding to the search query; The search results are written into the current inference trajectory as environmental observations, and the strategy model generates reflection content based on the inference trajectory after the search results are written. The reflection content is used to identify uncertain information, missing information or inference bias in the current evidence state, and to maintain, correct or redirect the current evidence acquisition target accordingly. When the reflection content indicates that retrieval needs to continue, the process of generating search queries, writing retrieval results, and generating reflection content is repeated until the preset stopping condition is met, and a complete reasoning trajectory is obtained. Based on the complete reasoning trajectory, the final answer is generated by the strategy model; A hierarchical verifiable reward is constructed based on the final answer, the reflection content, and the complete reasoning trajectory, and the hierarchical verifiable reward is used as the reward signal for the reinforcement learning algorithm; Based on the verifiable rewards in the hierarchical structure, the policy model is trained and updated using a reinforcement learning algorithm to obtain an enhanced reasoning model for target retrieval.

2. The method of claim 1, wherein, In the structured retrieval enhanced reasoning template: The plan label is used to constrain the strategy model to output a high-level evidence acquisition strategy; The search tags are used to constrain the strategy model to generate query statements for the external retrieval environment; The information tag is used to carry the environmental observation returned by the external retrieval environment, and the content corresponding to the information tag is the observation information returned by the external retrieval environment, which does not belong to the action space generated autonomously by the strategy model. The reflection label is used to constrain the strategy model to evaluate the current evidence status and update the current evidence acquisition target; The answer label is used to constrain the policy model to output a final answer based on accumulated evidence.

3. The method according to claim 1, characterized in that, The current reasoning trajectory includes at least: the problem to be reasoned, the initial plan, the generated historical search queries, the search results corresponding to each historical search query, the generated historical reflection content, and the generated intermediate reasoning content. The strategy model maintains the current evidence acquisition target based on the current inference trajectory, and dynamically updates the current evidence acquisition target based on newly added environmental observations.

4. The method of claim 1, wherein, The tiered verifiable rewards include at least exact match rewards, partial result rewards, reflective perception process rewards, and format rewards; The exact match reward is used to provide the highest level of reward when the final answer is exactly matched with the labeled answer; The partial result reward is used to provide a reward based on the degree of word-level matching when the final answer partially overlaps with the labeled answer; The reflection and perception process reward is used to provide a process reward when the final answer does not exactly match the labeled answer but the reflection content contains the labeled answer; and The format reward is used to provide a reward for reasoning trajectories that meet the preset structured label specifications.

5. The method of claim 1, wherein, The initial plan for generation includes: Generate a high-level reasoning strategy based on the problem to be reasoned about, which describes the multi-step solution path and the order of evidence acquisition. The high-level reasoning strategy is used to determine an initial chain of evidence before the first retrieval, so that subsequent rounds of retrieval are organized around the initial chain of evidence, rather than triggering retrieval based solely on local uncertainty.

6. The method of claim 1, wherein, The generated reflection content includes: Determine whether the currently obtained evidence is sufficient to support the output of the final answer; Identify uncertainties, missing evidence, or erroneous inferences in the current reasoning process; If there is insufficient evidence or the current search direction has shifted, determine the evidence slots to be completed and update the evidence acquisition targets corresponding to the subsequent searches based on the evidence slots to be completed.

7. The method according to claim 1, characterized in that, The preset stop condition includes at least one of the following: The reflections indicate that the current evidence is sufficient to support the output of the final answer; No new valid evidence was obtained in two consecutive rounds of searching; The preset maximum number of search rounds has been reached.

8. The method according to claim 1, characterized in that, The step of training and updating the policy model using a reinforcement learning algorithm based on the verifiable reward in the hierarchical manner includes: Multiple candidate reasoning trajectories are generated by sampling for the same problem to be reasoned; Calculate the stratified verifiable reward for each candidate reasoning trajectory; The relative advantage value within the group is determined by the stratified verifiable reward corresponding to each candidate reasoning trajectory. The strategy model is updated based on the relative advantage value within the group.

9. The method according to claim 8, characterized in that, The step of updating the parameters of the strategy model based on the relative advantage value within the group includes: The observation terms corresponding to the environmental observations returned by the external retrieval environment are masked so that the loss calculation is focused on the action terms generated by the policy model. A reference policy model is introduced during the training process to impose regularization constraints on the policy model, thereby suppressing policy drift during training.

10. A training device for a retrieval-enhanced reasoning model, characterized in that, include: The sample acquisition module is used to acquire a training sample set, which includes a question to be reasoned and labeled answers corresponding to the question to be reasoned. The template building module is used to build structured retrieval enhanced reasoning templates, which include at least plan tags, search tags, information tags, reflection tags, and answer tags; The plan generation module is used to input the problem to be reasoned into the policy model to be trained, so that the policy model generates an initial plan based on the structured retrieval enhanced reasoning template, wherein the initial plan is used to determine the initial evidence acquisition target for the problem to be reasoned; The retrieval interaction module is used to generate a search query from the strategy model based on the current inference trajectory, and send the search query to an external retrieval environment to obtain the retrieval results corresponding to the search query. The trajectory update module is used to write the retrieval results as environmental observations into the current inference trajectory, and based on the inference trajectory after writing the retrieval results, the strategy model generates reflection content, wherein the reflection content is used to identify uncertain information, missing information or inference bias in the current evidence state, and thereby maintain, correct or redirect the current evidence acquisition target. The reflection update module is used to repeatedly execute the search query generation, search result writing, and reflection content generation when the reflection content indicates that retrieval needs to continue, until the preset stopping condition is met, and the complete reasoning trajectory is obtained. The answer generation module generates the final answer based on the complete reasoning trajectory and the strategy model. The reward construction module is used to construct a hierarchical verifiable reward based on the final answer, the reflection content, and the complete reasoning trajectory, and to use the hierarchical verifiable reward as the reward signal for the reinforcement learning algorithm. The model training module is used to train and update the policy model using a reinforcement learning algorithm based on the hierarchical verifiable rewards, thereby obtaining a target retrieval enhanced reasoning model.

Images