A dynamic retrieval augmentation generation method based on reinforcement learning
By using a reinforcement learning-driven dynamic retrieval enhancement generation method, the problem of poor retrieval path selection in existing technologies is solved, achieving efficient and accurate answer generation and improving the accuracy and robustness of the question-answering system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YUNNAN NORMAL UNIV
- Filing Date
- 2026-02-05
- Publication Date
- 2026-06-09
AI Technical Summary
Existing retrieval enhancement generation methods lack dynamic optimization mechanisms, resulting in poor retrieval path selection, difficulty in integrating noisy information and non-parametric knowledge, and affecting the accuracy of question answering.
We adopt a dynamic retrieval enhancement generation method based on reinforcement learning. By training an open-source language model, it dynamically selects the optimal strategy among three paths: direct generation, single retrieval, and iterative retrieval. Combined with a knowledge compression model and a knowledge guidance model, we optimize the retrieval results.
It significantly improves the accuracy and reliability of question answering. Through effectiveness assessment, comparison of information content and relevance, it avoids interference from redundant information, dynamically optimizes retrieval queries, and enhances the factual correctness and interpretability of multi-hop question answering.
Smart Images
Figure CN121638478B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a dynamic retrieval enhancement generation method based on reinforcement learning, belonging to the field of natural language processing and large model retrieval enhancement generation. Background Technology
[0002] With the rapid development of internet technology, generative artificial intelligence models have played a crucial role in natural language understanding, information retrieval, and human-computer interaction. These models are widely applied in search engines, content generation, online customer service, and knowledge-based question answering, providing efficient solutions for information retrieval, personalized recommendations, and knowledge-based reasoning. In recent years, large-scale language models have received widespread attention for their superior performance in knowledge integration and reasoning capabilities; however, their inherent parameterized knowledge still has limitations. When problems involve up-to-date information or specialized domain knowledge, the model may generate seemingly reasonable but actually erroneous or non-existent content—a phenomenon known as the illusion problem.
[0003] To alleviate the hallucination problem, researchers have proposed the Retrieval Augmented Generation (RAG) method. This technique enables the mounting of an external knowledge base, through which non-parametric knowledge (usually text) related to the question is retrieved. By inputting the question and non-parametric knowledge as prompts into a large model, the model completes question-answering with the assistance of additional knowledge, ultimately improving its accuracy. However, existing RAG methods face several key technical challenges: the lack of dynamic optimization mechanisms for retrieval path selection, the presence of noisy or irrelevant information and non-parametric knowledge in retrieved documents, and the significant difficulty in fusing parametric knowledge. Summary of the Invention
[0004] The purpose of this invention is to provide a dynamic retrieval enhancement generation method based on reinforcement learning, which aims to solve the technical problem that existing technologies lack a dynamic optimization mechanism for retrieval path selection.
[0005] To achieve the above objectives, the technical solution of this invention is: a dynamic retrieval enhancement generation method based on reinforcement learning. This method trains an open-source language model using reinforcement learning, enabling it to dynamically select the optimal strategy among three paths: direct generation, single-step retrieval, and iterative retrieval. Specifically, the single-step retrieval path incorporates a knowledge compression model to extract concise documents, while the iterative retrieval path incorporates a knowledge-guided model to generate inference chains and progressively optimize retrieval results. This effectively alleviates the illusion problem of large models through the synergistic effect of self-knowledge and external knowledge, improving the accuracy and reliability of question answering. The specific steps are as follows:
[0006] Step 1: Construct a training dataset based on the classification prompt word template and the original dataset, and use the training dataset to train the open-source language model to obtain the generative model;
[0007] Step 2: Input the user-inputted question text into the generation model and output the optimal retrieval enhancement generation path; wherein, the optimal retrieval enhancement generation path includes a direct generation path, a single retrieval path, and an iterative retrieval path;
[0008] Step 3: Generate the final answer to the question text based on the optimal search enhancement generation path;
[0009] When the generated path is the direct generation path, the generation model generates candidate answers, and evaluates the effectiveness, information content and relevance of the candidate answers to generate the final answer.
[0010] Specifically, when the generated path is the single retrieval path, document retrieval and knowledge compression are performed to extract concise documents, and the final answer is generated based on the concise documents;
[0011] When the generated path is the iterative retrieval path, the retrieval query is dynamically optimized, and the final answer is generated after integrating multiple rounds of retrieval information.
[0012] Optionally, Step 1 specifically includes:
[0013] Step 1.1: Set up classification prompt word templates for problem complexity assessment, and define the original dataset as follows. In the form of question-and-answer pairs, For the question, To answer the question The classification prompt word template is input into an external large model, and the classification result is output. Thus, the training dataset is formed. ;
[0014] Step 1.2: Based on the training dataset The generative model is trained using reinforcement learning. During the training process, a lightweight pre-trained language model is used to encode the input question, and a fine-grained reward function is used to update the parameters of the open-source language model to obtain the generative model.
[0015] Optionally, the fine-grained reward function expression is:
[0016]
[0017] in, For fine-grained rewards, Indicates the path The generated answer, This indicates the degree of match between the generated answer and the actual answer. Indicates computational cost. This is the balance coefficient.
[0018] Optionally, when the generated path is the direct generation path, the generation model generates candidate answers, and the candidate answers are evaluated for effectiveness, information content, and relevance to generate the final answer. Specifically, this process involves:
[0019] A prompt word template is set to generate candidate answers and answer explanation summaries. The question text under the direct generation path and the prompt word template are input into the generation model to obtain candidate answers and corresponding explanation summaries.
[0020] A prompt word template for effectiveness assessment and comparison of information content and relevance is defined. The candidate answers and their corresponding explanatory summaries are input into the generation model along with the prompt word template for effectiveness assessment and comparison of information content and relevance to calculate the results for each candidate answer. Corresponding Explanation Summary Validity score and comparative scores The expression is:
[0021]
[0022] in, This indicates the problem text. Used to evaluate each summary The effectiveness;
[0023]
[0024] in, Indicates the first Explanation summary of each candidate answer, Used to compare all explanatory summaries middle and Information content and relevance;
[0025] Calculate the total score The expression is:
[0026]
[0027] in, Represents the total number of explanatory summaries;
[0028] Get the final answer ,in, .
[0029] Optionally, when the generated path is the single retrieval path, performing document retrieval and knowledge compression, extracting concise documents, and generating the final answer based on the concise documents specifically involves:
[0030] The knowledge compression model is integrated with the generation probability of an open-source language model, and the token with the highest probability is selected as the output, thereby completing the training of the knowledge compression model.
[0031] Retrieve Search Documents The retrieved document is input into the trained knowledge compression model to obtain a concise document. Then, the concise document... With the question The input is fed into the generative model to obtain the final answer, expressed as:
[0032]
[0033] in, To generate the conditional probability distribution of the model, This indicates the final answer.
[0034] Optionally, when the generated path is the iterative retrieval path, dynamically optimizing the retrieval query and integrating multi-round retrieval information to generate the final answer specifically involves:
[0035] Construct a training dataset for reasoning chains. Each reasoning chain consists of reasoning, actions, and observation results. During training, reasoning and actions are generated through instruction prompts. If the action is "search", the retrieval is performed to obtain the observation results and append them to the reasoning chain.
[0036] Based on the aforementioned inference chain training dataset, the open-source language model is subjected to supervised fine-tuning to obtain a knowledge-guided model. This supervised fine-tuning involves maximizing the conditional probability of generating inference chains.
[0037]
[0038] in, Indicates the first One question, Represents the reasoning chain Length, This represents the probability distribution of the knowledge-guided model. Let be the conditional probability of each inference chain. The reasoning-action-observation triple is used for reasoning and thinking. This is a historical reasoning process;
[0039] The knowledge-guided model dynamically generates reasoning and actions. If the action is "search," a retrieval is performed to obtain observation results. These observation results, along with the historical reasoning chain, are used as input for the next round of reasoning. When the iteration reaches the point where the model selects the "finish" action, the complete reasoning chain is used. The final answer is generated by integrating information; the expression is:
[0040]
[0041] in, To generate the conditional probability distribution of the model, This indicates the final answer.
[0042] The beneficial effects of this invention are as follows: This invention, through a reinforcement learning-driven dynamic retrieval enhancement generation method, can achieve efficient answer generation under three paths: direct generation, single-step retrieval, and iterative retrieval. Specifically, the direct generation path introduces validity evaluation of candidate answers and comparisons of information content and relevance, improving answer reliability. The single-step retrieval path trains a knowledge compression model, compressing the retrieved documents into core content highly relevant to the question, and enhances the distribution alignment with the generation model through probability distribution weighted integration, thereby avoiding redundant information interference and improving answer relevance. The iterative retrieval path proposes a knowledge-guided reasoning chain mechanism, dynamically optimizing the retrieval query and gradually integrating evidence documents in the form of "reasoning-action-observation result" triples, significantly enhancing the factual correctness and interpretability of multi-hop question answering. Overall, this invention outperforms existing methods in terms of accuracy, efficiency, and robustness, and has strong practical value. Attached Figure Description
[0043] Figure 1 This is a flowchart of the steps of the method of the present invention. Detailed Implementation
[0044] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.
[0045] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0046] Example 1: As Figure 1 As shown, a dynamic retrieval enhancement generation method based on reinforcement learning includes the following steps:
[0047] Step 1: Construct a training dataset based on the classification prompt word template and the original dataset, and use the training dataset to train the open-source language model to obtain the generative model;
[0048] Step 1.1: Set up classification prompt word templates for problem complexity assessment, and define the original dataset as follows. In the form of question-and-answer pairs, For the question, To answer the question The classification prompt word template is input into an external large model, and the classification result is output. Thus, the training dataset is formed. ;
[0049] Optionally, in this embodiment, the external large model can be a GPT series model (closed source), and this embodiment uses GPT-4.
[0050] Specifically, in this embodiment, the classification prompt word template used for problem complexity assessment is:
[0051] Your task is to act as a classification expert. Please determine the question category and output the result based on the following criteria: Direct answer: The question is simple and can be answered based on common sense, without the need for searching. Single search: The question is of medium difficulty and can be answered with a single search. Complex question: The question requires multiple steps of reasoning or integration of information from multiple sources to answer.
[0052] question:{}
[0053] Output: {}";
[0054] Step 1.2: Based on the training dataset The generative model is trained using reinforcement learning. During the training process, a lightweight pre-trained language model is used to encode the input question, and a fine-grained reward function is used to update the parameters of the open-source language model to obtain the generative model.
[0055] Optionally, in this embodiment, the reinforcement learning adopts the PPO algorithm, and the open-source language model can be a series of models such as BERT, GPT2 and LLaMA. In this embodiment, LLaMA3-8B is adopted.
[0056] Optionally, in this embodiment, the lightweight pre-trained language model can be DistilBERT, which is used to address the problem. and its classification results Generate action distribution vector ,in, DistilBERT is a lightweight pre-trained language model, and its distribution vectors are shown below. Each element in the table represents a predicted score for a path. Selected path The method with the highest prediction score, i.e. The specific paths are divided into three types: direct generation path, single search path, and iterative search path. For the selected path... The fine-grained reward function expression is as follows:
[0057]
[0058] in, For fine-grained rewards, Indicates the path The generated answer, This indicates the degree of match between the generated answer and the actual answer. Indicates computational cost. This is the balance coefficient.
[0059] Furthermore, the optimization objective of the generative model is to minimize the following loss function, expressed as:
[0060]
[0061] The model parameters are updated using gradients based on the fine-grained reward function, expressed as follows:
[0062]
[0063] in, for Model parameters for the next iteration for Model parameters for the next iteration For the parameters of the model, Indicates the parameter gradient, The learning rate;
[0064] It is understandable that, by minimizing the loss function, the generative model can dynamically select the retrieval enhancement generation path based on the input question and its complexity.
[0065] Step 2: Input the user-inputted question text into the generation model and output the optimal retrieval enhancement generation path; wherein, the optimal retrieval enhancement generation path includes a direct generation path, a single retrieval path, and an iterative retrieval path;
[0066] Step 3: Generate the final answer to the question text based on the optimal search enhancement generation path;
[0067] Specifically, when the generated path is the direct generation path, the generation model generates candidate answers, and the candidate answers are evaluated for effectiveness, information content and relevance, and the final answer is generated.
[0068] Optionally, a prompt word template is set for generating candidate answers and answer explanation summaries. The question text under the direct generation path and the prompt word template are input into the generation model to obtain candidate answers and corresponding explanation summaries.
[0069] Specifically, in this embodiment, a prompt word template is used to generate candidate answers and answer explanation summaries. and for:
[0070] :"
[0071] Imagine you have read many articles on different topics. Now, based on this general knowledge, please provide two correct answer options for the following questions. Each answer should be presented in the following form: (1) xx, (2) yy, with each option consisting of no more than 3 words.
[0072] refer to:{}
[0073] question:{}
[0074] Answer:";
[0075] :"
[0076] refer to: {}
[0077] Your task is to act as a professional writer. You need to write a high-quality article to support a given prediction about the issue, based on the information provided in the supporting paragraphs. Now, let's begin. When you're finished, write "[Done]" to indicate that you're done. Do not add prefixes (e.g., "Response:") when writing your article.
[0078] question:{}
[0079] predict:{}
[0080] article:";
[0081] Furthermore, prompt word templates are set for effectiveness assessment and comparison of information content and relevance;
[0082] Specifically, in this embodiment, the prompt word template is used for effectiveness assessment and comparison of information content and relevance. and for:
[0083] :"
[0084] question:{}
[0085] predict:{}
[0086] article:{}
[0087] Does the article correctly support the prediction? Select: [Correct, Incorrect].
[0088] Answer:";
[0089] :"
[0090] refer to: {}
[0091] Question: Based on the following articles, determine which one provides the most informative answer to the subsequent question.
[0092] Article 1: {}
[0093] Article 2: {}
[0094] Target problem: {}
[0095] Your task:
[0096] Identify which article (Article 1 or Article 2) is more relevant to the user's question and contains more information. Select: [Article 1, Article 2].
[0097] Answer:";
[0098] Furthermore, the candidate answers and their corresponding explanatory summaries, along with the prompt word templates used for validity assessment, information content comparison, and relevance comparison, are input into the generation model to calculate each candidate answer. Corresponding Explanation Summary Validity score and comparative scores The expression is:
[0099]
[0100] in, This indicates the problem text. Used to evaluate each summary The effectiveness;
[0101]
[0102] in, Indicates the first Explanation summary of each candidate answer, Used to compare all explanatory summaries middle and The amount of information and relevance, if Superior , 1 point; if Superior , 0 points; in other cases, 0.5 points;
[0103] Calculate the total score The expression is:
[0104]
[0105] in, Represents the total number of explanatory summaries;
[0106] Furthermore, based on and The candidate answer with the highest score will be the final answer. ,in, .
[0107] Specifically, when the generated path is the single retrieval path, document retrieval and knowledge compression are performed to extract concise documents, and the final answer is generated based on the concise documents;
[0108] The knowledge compression model is integrated with the generation probability of an open-source language model, and the token with the highest probability is selected as the output to complete the training of the knowledge compression model. The expression is as follows:
[0109]
[0110] in, For concise documentation, related to height input. For the collection of retrieved documents, This represents the probability distribution of the knowledge compression model.
[0111] Furthermore, in the decoding phase, the final token is obtained through weighted integration, expressed as:
[0112]
[0113] in, This represents the currently generated token. This represents a vocabulary containing all possible tokens. express It is a vocabulary list One of the elements, For integration coefficients, and These are knowledge compression instructions and context generation instructions, respectively. , These represent the probability distributions of the knowledge compression model and the generation model on the next token, respectively. Indicates the preceding i -1 generated token, To retrieve documents.
[0114] Optionally, in this embodiment, the knowledge compression model may be RECOMP.
[0115] It is understood that this embodiment completes the training of the knowledge compression model through the above integration method, which makes the generated context more consistent with the distribution characteristics of the generated model.
[0116] Furthermore, the retrieved document is obtained and input into the trained knowledge compression model to obtain a concise document. Then, the concise document... With the question The input is fed into the generative model to obtain the final answer, expressed as:
[0117]
[0118] in, To generate the conditional probability distribution of the model, This indicates the final answer.
[0119] Specifically, when the generated path is the iterative retrieval path, the retrieval query is dynamically optimized, and the final answer is generated after integrating information from multiple rounds of retrieval.
[0120] Furthermore, a training dataset for inference chains is constructed based on the multi-hop problem. Each inference chain consists of reasoning, actions, and observation results. During training, reasoning and actions are generated through instruction prompts. If the action is "search", the retrieval is performed to obtain the observation results and append them to the inference chain.
[0121] The open-source language model is supervised and fine-tuned based on the inference chain training dataset, enabling it to generate inference-action-observation triples at the inference chain level. The ability to generate a knowledge-guided model is used to obtain the knowledge-guided model, and the supervised fine-tuning is to maximize the conditional probability of generating the inference chain.
[0122]
[0123] in, Indicates the first One question, Represents the reasoning chain The length of the inference chain, i.e., the number of triples in the inference chain. This represents the probability distribution of the knowledge-guided model. Let be the conditional probability of each inference chain. This is a historical reasoning process;
[0124] Using knowledge-guided models to dynamically generate reasoning, thinking, and actions, based on user questions. Generate an empty inference chain The knowledge-guided model generates the next reasoning and action based on the current problem and the historical reasoning chain. ):
[0125]
[0126] If the action is "search", then perform the search and obtain the observation results. and will ( This is added to the inference chain for the next round of inference. Iteration stops when the model selects the "finish" action. (Based on the complete inference chain) The final answer is generated by integrating information; the expression is:
[0127]
[0128] in, To generate the conditional probability distribution of the model, This indicates the final answer.
[0129] It is important to understand that in this embodiment, search and finish are generated by the knowledge-guided model based on the current question and the historical reasoning chain. When the observation in the historical reasoning chain (triple) does not contain the final answer, the knowledge-guided model generates search and continues to search for relevant documents; if it already contains the final answer, finish is generated, indicating that the answer has been completed and no further iteration is needed.
[0130] Furthermore, to verify the effectiveness of this invention, comparative experiments were conducted on four representative question-answering datasets: NQ, TriviaQA, HotpotQA, and PopQA, using exact matching (EM) and F1 score as evaluation metrics. In the experiments, this embodiment selected two existing retrieval augmentation generation methods as benchmarks: Naive RAG and Rea-RAG. Naive RAG is a standard retrieval augmentation generation algorithm with 8B parameters; Rea-RAG is a retrieval augmentation generation algorithm incorporating reinforcement learning, with 9B parameters. By comparing with these two existing methods, the performance improvement of the method in this invention can be more clearly demonstrated. The experimental results are shown in Table 1.
[0131] Table 1 Performance comparison of the method of the present invention with existing retrieval enhancement generation methods
[0132]
[0133] The data in Table 1 show that the method of this invention exhibits superior performance on different types of question-answering datasets. Although the number of model parameters is smaller than that of the ReaRAG method, it outperforms existing retrieval enhancement generation methods in terms of both accurate matching and answer generation quality when considering all indicators.
[0134] In summary, this invention employs reinforcement learning to train a generative model, achieving dynamic path selection. The paths include three types: first, a direct generation path, where a large model generates candidate answers and outputs the final answer based on validity assessment, information content comparison, and relevance comparison; second, a single-retrieval path, where knowledge compression is applied to the retrieved documents, extracting concise documents highly relevant to the question to generate the final answer; and third, an iterative retrieval path, where a knowledge-guided model dynamically optimizes the retrieval query, integrating multi-round retrieval information to generate the final answer. This invention, through a dynamic path selection mechanism combined with knowledge compression and guidance strategies, effectively solves the problems of excessive retrieval, erroneous retrieval, and insufficient reasoning in traditional fixed-process systems, significantly improving the accuracy, robustness, and response efficiency of large-scale model question-answering systems.
[0135] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.
Claims
1. A dynamic retrieval enhancement generation method based on reinforcement learning, characterized in that, The method includes the following steps: Step 1: Construct a training dataset based on the classification prompt word template and the original dataset, and use the training dataset to train the open-source language model to obtain the generative model; Step 2: Input the user-inputted question text into the generation model and output the optimal retrieval enhancement generation path; wherein, the optimal retrieval enhancement generation path includes a direct generation path, a single retrieval path, and an iterative retrieval path; Step 3: Generate the final answer to the question text based on the optimal search enhancement generation path; When the generated path is the direct generation path, the generation model generates candidate answers, and evaluates the effectiveness, information content and relevance of the candidate answers to generate the final answer. Specifically, when the generated path is the single retrieval path, document retrieval and knowledge compression are performed to extract concise documents, and the final answer is generated based on the concise documents; When the generated path is the iterative retrieval path, the retrieval query is dynamically optimized, and the final answer is generated after integrating multiple rounds of retrieval information. Step 1 specifically refers to: Step 1.1: Set up classification prompt word templates for problem complexity assessment, and define the original dataset as follows. In the form of question-and-answer pairs, For the question, To answer the question The classification prompt word template is input into an external large model, and the classification result is output. Thus, the training dataset is formed. ; Step 1.2: Based on the training dataset The generative model is trained using reinforcement learning. During the training process, a lightweight pre-trained language model is used to encode the input question, and a fine-grained reward function is used to update the parameters of the open-source language model to obtain the generative model. The fine-grained reward function expression is as follows: ； in, For fine-grained rewards, Indicates the path The generated answer, This indicates the degree of match between the generated answer and the actual answer. Indicates computational cost. This is the balance coefficient; When the generated path is the single retrieval path, document retrieval and knowledge compression are performed to extract concise documents, and the final answer is generated based on the concise documents. Specifically, this involves: The knowledge compression model is integrated with the generation probability of an open-source language model, and the token with the highest probability is selected as the output, thereby completing the training of the knowledge compression model. Retrieve Search Documents The retrieved document is input into the trained knowledge compression model to obtain a concise document. Then, the concise document... With the question The input is fed into the generative model to obtain the final answer, expressed as: ； in, To generate the conditional probability distribution of the model, This indicates the final answer.
2. The dynamic retrieval enhancement generation method based on reinforcement learning according to claim 1, characterized in that, When the generated path is the direct generation path, the generation model generates candidate answers, and the candidate answers are evaluated for effectiveness, information content, and relevance to generate the final answer. Specifically, this process is as follows: A prompt word template is set to generate candidate answers and answer explanation summaries. The question text under the direct generation path and the prompt word template are input into the generation model to obtain candidate answers and corresponding explanation summaries. A prompt word template for effectiveness assessment and comparison of information content and relevance is defined. The candidate answers and their corresponding explanatory summaries are input into the generation model along with the prompt word template for effectiveness assessment and comparison of information content and relevance to calculate the results for each candidate answer. Corresponding Explanation Summary Validity score Comparative scores The expression is: ； in, Indicates the problem text. Used to evaluate each summary The effectiveness; ； in, Indicates the first Explanation summary of each candidate answer, Used to compare all explanatory summaries middle and Information content and relevance; Calculate the total score The expression is: ； in, Represents the total number of explanatory summaries; Get the final answer ,in, .
3. The dynamic retrieval enhancement generation method based on reinforcement learning according to claim 1, characterized in that, When the generated path is the iterative retrieval path, the dynamic optimization of the retrieval query and the generation of the final answer after integrating multiple rounds of retrieval information specifically involve: Construct a training dataset for reasoning chains. Each reasoning chain consists of reasoning, actions, and observation results. During training, reasoning and actions are generated through instruction prompts. If the action is "search", the retrieval is performed to obtain the observation results and append them to the reasoning chain. Based on the aforementioned inference chain training dataset, the open-source language model is subjected to supervised fine-tuning to obtain a knowledge-guided model. This supervised fine-tuning involves maximizing the conditional probability of generating inference chains. ； in, Indicates the first One question, Represents the reasoning chain Length, This represents the probability distribution of the knowledge-guided model. Let be the conditional probability of each inference chain. The reasoning-action-observation triple is used for reasoning and thinking. This is a historical reasoning process; The knowledge-guided model dynamically generates reasoning and actions. If the action is "search," a retrieval is performed to obtain observation results. These observation results, along with the historical reasoning chain, are used as input for the next round of reasoning. When the loop iterates to the point where the model selects the "finish" action, it proceeds based on the complete reasoning chain. The final answer is generated by integrating information; the expression is: ； in, To generate the conditional probability distribution of the model, This indicates the final answer.