A PPO-based retrieval enhancement model construction method

Abstract

The application provides a PPO-based retrieval enhancement model construction method, an S-RAG retrieval model composed of a searcher LLM (S) and a generator LLM (G) is constructed, the searcher LLM (S) is optimized through reinforcement learning, and a retrieval enhancement gain GFR is used as a reward signal; a multi-round search-selection cycle is performed to retrieve relevant documents and construct a final context, the final context is provided to the frozen generator LLM (G), a final output is generated, the modularity separation of retrieval and generation is realized, the generality and flexibility of the framework are improved, the GFR reward is defined by comparing the performance improvement of the generator when the S-RAG retrieval document is used and when the simple top-k retrieval document is used, the limitation of the full matching reward is effectively overcome, the robustness of the retrieval enhancement gain reward signal is improved, the retrieval quality is improved, the training cost is greatly reduced, the training efficiency is improved, and the balance between retrieval precision and efficiency is realized.

Description

Technical Field

[0001] This invention relates to the field of retrieval enhancement generation for large language models, and in particular to a method for constructing a retrieval enhancement model based on PPO. Background Technology

[0002] Retrieval-enhanced generation (RAG) technology has significantly improved the generation accuracy and reliability of large language models (LLMs) in knowledge-intensive tasks by providing them with the ability to retrieve external knowledge, and has become a key technology for expanding the application scope of LLMs.

[0003] Early RAG methods used fixed query statements to obtain relevant knowledge from databases through static retrieval and then fed it into a large language model to generate answers. This simple system usually performed poorly on complex problems. Active RAG techniques [2–4] interleave query generation, retrieval, and reasoning in a multi-round loop, all of which introduce iterative retrieval, but they usually rely on zero-shot hints and lack trainable components.

[0004] In recent years, reinforcement learning has been introduced into the RAG field to optimize the retrieval and generation processes. Existing techniques, such as Search-R1, jointly train the retrieval and generator through end-to-end reinforcement learning, using perfect matching (EM) as the reward signal, which improves the answer accuracy to some extent. However, existing methods have the following drawbacks: First, the perfect matching reward lacks robustness and cannot provide effective feedback for semantically correct but differently expressed answers, limiting the upper limit of model optimization; second, the tight coupling between retrieval and generation components leads to high model training costs, and the trained retrieval unit is difficult to independently serve other black-box LLMs; finally, joint training easily leads to the generator overfitting to answer labels rather than truly improving retrieval quality, and existing models do not distinguish between problems that can be solved by simple top-k retrieval and difficult problems that require multiple rounds of complex retrieval. During training, a large amount of resources still need to be invested in simple problems, reducing training efficiency and failing to specifically improve the model's ability to handle complex problems. Summary of the Invention

[0005] The main objective of this invention is to provide a method for constructing a retrieval enhancement model based on PPO, which solves the technical problems of insufficient robustness of reward signals, excessive module coupling, high training cost, and redundant resource consumption in existing technologies.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by this invention is: a method for constructing a retrieval enhancement model based on PPO, wherein the searcher and generator are decoupled, and the retrievaler is trained by reinforcement learning while keeping the generator frozen, including the following steps: S1: Construct an S-RAG retrieval model consisting of a search engine LLM(S) and a generator LLM(G), wherein the search engine LLM(S) is modularly separated from the generator, and the generator LLM(G) remains frozen during training. S2: Use the search engine SerpApi to retrieve the first k relevant documents D0 based on the initial question Q, form the initial context, and then pass it into the generator G to obtain the initial answer; S3: The problem of rewriting the query based on Q using the search engine LLM(S) ; S4: Search engine SerpAPI based on The search retrieved the top k relevant documents. ; S5: Searcher S retrieves documents from a new round. Select important documents to form context And determine whether enough information has been found to stop the loop; Steps 3 through 5 are performed alternately to form a multi-round search-selection loop; S6: Combine the initial problem Q with the important documents obtained from the multi-round search-selection loop. The context is provided to the freeze generator LLM(G), which generates the output and calculates the retrieved enhancement gain GFR based on the returned result; S7: Implement reinforcement learning of the searcher LLM(S) based on the Proximal Policy Optimization (PPO) algorithm, where the reward signal is GFR; S8: Test the S-RAG retrieval model, which consists of the trained search engine LLM(S) and the frozen generator LLM(G), on another dataset and compare it with the current RAG retrieval scheme.

[0007] In the preferred embodiment, before performing the multi-round search-selection loop, the following steps are included: Initialization operation: Given a problem Q, use For the initial query, the first k documents to be tested are retrieved. Then select a subset This constitutes the initial context.

[0008] In the preferred embodiment, the multi-round search-selection loop includes the following sub-steps: In each iteration of the loop, the following steps are executed sequentially: The search engine generates a query and return the results in a preset query format; According to the query Search related documents And return the results in a preset information format; Based on the prompts, retrieve documents Select a useful subset of relevant documents And return the results in a preset selection format; When no new documents are updated or the round limit is reached, output a declaration message in a preset end format and the loop ends; After the loop terminates, the final context is obtained and passed to the frozen generator to generate the final output.

[0009] In the preferred embodiment, in step S3, the definition of the retrieval enhancement gain GFR is: ; in, As the standard answer, Retrieved from the original question One result, For a task-specific metric, GenAcc (Generated Accuracy) is used, expressed as: ; In the formula, For rapid span matching testing, For correctness checks based on lightweight LLM.

[0010] In a preferred embodiment, a sample selection step is further included before training the search engine: Selecting training samples: Pre-calculate the baseline accuracy term Acc(G(Q,D_RAG),A), filter out samples with a baseline accuracy term equal to 0, and only keep queries that are difficult to solve with simple RAG as training samples.

[0011] In the preferred embodiment, the reinforcement learning employs a proximal strategy to optimize the PPO algorithm, including: Using GFR rewards, the searcher's search strategy is optimized through reinforcement learning, with each deployment including a complete search trajectory: the issued query, document selection, and stopping decision; once the final context is built... generator This will generate the answer and calculate the GFR reward, and the generator will be optimized using the PPO algorithm; The objective function of PPO is: ; in, This represents the probability ratio between the current strategy and the reference strategy. For the estimated advantage, This is the clipping threshold.

[0012] In the preferred embodiment, during the reinforcement learning training process, the generator remains frozen, and the gradient is backpropagated only through the searcher's strategy.

[0013] In the preferred embodiment, the training of the searcher and freeze generator models in step S1 further includes: Qwen2.5-7B-Instruct was selected as the base searcher LLM for training; Qwen2.5-14B-Instruct was selected as the frozen generator for answer generation and the frozen scorer for judge_check; A training set was constructed by combining samples from Natural Question Answering (NQ) and HotpotQA. All yes / no and true / false questions were removed from the training set. NQ, TriviaQA, PopQA, HotpotQA, and 2wiki were selected as test datasets. Among them, NQ, TriviaQA, and PopQA are single-hop inference datasets, while HotpotQA and 2wiki are multi-hop inference datasets. Training parameter settings: During training, the number of training samples was set to 2.4k, the batch size to 120, and the number of training steps to 20 PPO steps.

[0014] In the preferred embodiment, the scoring criteria for document selection are determined based on the relevance of the document to the query, the completeness of the content, and its contribution to generating the correct answer. The document selection follows the following rules: newly added documents are marked as important, documents without important tags are deduplicated, and a new round of document screening is obtained by combining the two. The number of important documents selected each time does not exceed 3.

[0015] In the preferred embodiment, the search engine LLM(S) and the generator LLM(G) interact using preset prompts, wherein: the prompts of the search engine are used to guide it to complete the logical execution of query generation, document selection and loop termination judgment; The generator's prompts are used to guide users to answer questions directly based on the provided context, without including any extraneous text.

[0016] This invention provides a method for constructing a retrieval enhancement model based on PPO. It constructs an S-RAG retrieval model consisting of a searcher and a generator. The searcher is optimized through reinforcement learning, and its reward signal is the retrieval enhancement gain (GFR). Multiple rounds of search-selection loops are performed to retrieve relevant documents and construct the final context. The final context is provided to the frozen generator to generate the final output. This achieves modular separation of retrieval and generation, improving the framework's versatility and flexibility. The GFR reward is defined by comparing the performance improvement of the generator when using S-RAG to retrieve documents with that of using simple top-k document retrieval. This effectively overcomes the limitations of perfect match rewards, improves the robustness of the retrieval enhancement gain reward signal, enhances retrieval quality, significantly reduces training costs, improves training efficiency, and achieves a balance between retrieval accuracy and efficiency. Attached Figure Description

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of the construction method of the present invention; Figure 2 This is the reward curve for the top-k={3,5,8} of this invention. Detailed Implementation

[0018] Example 1 like Figure 1-2 As shown, a method for constructing a retrieval enhancement model based on PPO includes the following steps: S1: Construct an S-RAG retrieval model consisting of a searcher LLM(S) and a generator LLM(G), wherein the searcher LLM(S) and the generator are modularly separated, and the generator LLM(G) is kept frozen during training.

[0019] S2: Use the search engine SerpApi to retrieve the first k relevant documents D0 based on the initial question Q, form the initial context, and then pass it into the generator G to obtain the initial answer.

[0020] S3: The problem of rewriting the query based on Q using the search engine LLM(S) .

[0021] S4: Search engine SerpAPI based on The search retrieved the top k relevant documents. .

[0022] S5: Searcher S retrieves documents from a new round. Select important documents to form context It then determines whether enough information has been found to stop the loop.

[0023] Steps S3 to S5 are performed alternately to form a multi-round search-selection loop.

[0024] S6: Combine the initial problem Q with the important documents obtained from the multi-round search-selection loop. The context is provided to the freeze generator LLM(G), which generates the output and calculates the retrieved enhancement gain GFR based on the returned result.

[0025] S7: Implement reinforcement learning of the searcher LLM(S) based on the Proximal Policy Optimization (PPO) algorithm, where the reward signal is GFR.

[0026] S8: Test the S-RAG retrieval model, which consists of the trained search engine LLM(S) and the frozen generator LLM(G), on another dataset and compare it with the current RAG retrieval scheme.

[0027] This embodiment constructs an S-RAG retrieval model consisting of a searcher LLM(S) and a generator LLM(G). The searcher LLM(S) is optimized through reinforcement learning, and its reward signal is the retrieval enhancement gain (GFR). Multiple rounds of search-selection loops are performed to retrieve relevant documents and construct the final context. The final context is provided to the frozen generator LLM(G) to generate the final output. This achieves modular separation of retrieval and generation, improving the framework's versatility and flexibility. The GFR reward is defined by comparing the performance improvement of the generator when using S-RAG to retrieve documents with that when using simple top-k document retrieval. This effectively overcomes the limitations of perfect match rewards, improves the robustness of the retrieval enhancement gain (GFR) reward signal, enhances retrieval quality, significantly reduces training costs, improves training efficiency, and achieves a balance between retrieval accuracy and efficiency.

[0028] This embodiment proposes a modular framework, namely the S-RAG retrieval model. Through three main approaches—module separation design, precise reward signals, and targeted training optimization—it effectively addresses the shortcomings of existing technologies, achieving significant improvements in performance, efficiency, and flexibility. The search and downstream LLM generation are separated, achieving a decoupled design between retrieval and generation. Reinforcement learning optimizes only the search quality, avoiding overfitting of the generator to answer labels and directly focusing on retrieval quality optimization. Furthermore, it allows the trained retrieval unit to flexibly adapt to any black-box LLM, greatly enhancing the framework's versatility and flexibility. S-RAG uses the reward signal—Retrieval Enhancement Gain (GFR)—for reinforcement learning (RL). By comparing the generator's performance with documents retrieved by S-RAG as conditions, it improves training performance compared to simple top-k retrieval. The generator's LLM is kept frozen to avoid answer label overfitting, and the retrieval component is directly optimized, enabling it to serve any black-box LLM.

[0029] Meanwhile, by leveraging the PPO algorithm and multi-round search-selection loops, performance verification was completed on the test dataset, improving the reliability of the model optimization effect and achieving a balance between retrieval accuracy and training efficiency.

[0030] I. After constructing the modular framework S-RAG in this embodiment, a baseline is first specified. The improvement in the accuracy of the retrieved context generation relative to the baseline is used as a reward to optimize the search module, thereby specifically improving the retrieval quality and separating it from answer generation. The specific principles of the S-RAG retrieval model are introduced first, and the improvement effect of the S-RAG retrieval model compared to the traditional RAG method is demonstrated.

[0031] II. Performing a multi-round search - selection loop: Initialization: Given a problem Q, the searcher LLM... and freeze generator LLM )composition. First use Retrieve the first k documents. Initialization ensures that the loop starts with the same context as a simple RAG, thus reflecting the search improvement of the current module compared to a general RAG.

[0032] In the preferred scheme, the value range of the number k in the documents to be tested is 3-8.

[0033] Searcher LLM Obtained through retrieval Then select a subset This forms the initial context. The system then loops through and executes a series of search operations. The specific algorithm structure is shown in Table 1: Table 1. Multi-round search loop algorithm

[0034] The loop continues until... Or the round limit is reached, resulting in the final context as follows: The context is passed to the generator to produce the final output: .

[0035] This embodiment employs a standardized multi-round search-selection loop process, standardizing the search engine's query generation, document retrieval, important document filtering, and loop termination criteria, thereby improving the model's stability and reproducibility. Leveraging the SerpAPI search engine enhances the comprehensiveness of document retrieval, accurately extracting useful information through a subset filtering mechanism, and effectively avoiding invalid loops by combining termination conditions (no new document updates or round limitations), thus improving retrieval accuracy and reducing computational costs.

[0036] III. Training via Gain Signal To train the search engine LLM The search is treated as a reinforcement learning (RL) problem, and the reward signal is named the search enhancement gain (GFR). The GFR quantizes the reward relative to a fixed value. The improvement in accuracy of baseline generation is expressed by the following formula: ; in As the standard answer, Retrieved from the original question One result, As a task-specific metric, this embodiment primarily uses generation accuracy to measure performance, combining rapid span matching tests and correctness checks based on lightweight LLM.

[0037] Given a model prediction and a set of standard answers And calculate: ; In the formula, For rapid span matching testing, This is a correctness check based on a lightweight LLM; the fast span matching test performs a regular expression match between the segments in p and A, and returns 1 (positive example) if a match is found, otherwise returns 0 (negative example); the correctness check uses the prompt word "Does p contain A?" to determine whether the result is correct or not through the results returned by the lightweight LLM.

[0038] Compared to traditional exact matching, the current metric not only considers the degree of field matching, but also compares the span of different fields in the sentence to infer semantic correctness; it uses a language model to judge semantic similarity, and is not affected by the superficial similarity of the answers.

[0039] Selecting training samples: Pre-calculating the baseline accuracy term The algorithm filters out samples with a baseline accuracy of 0, meaning it removes problems that can be solved with a simple RAG algorithm, and retains the difficult-to-solve queries as training samples. In other words, it filters out problems that can already be solved with a simple RAG algorithm, thus improving the LLM search engine's performance. It allows us to focus on more difficult queries, thus improving training efficiency.

[0040] IV. Search Strategy Optimization This embodiment utilizes GFR rewards to optimize the search engine through reinforcement learning. Each deployment includes a complete search trajectory: the query issued, document selection, and stopping decision. Once the final context is built... generator This generates an answer and calculates the GFR reward. During this process, the generator remains frozen, and gradients are backpropagated only through the search policy. The method in this embodiment is independent of any specific advantage estimation algorithm; it uses Population Relative Policy Optimization (PPO) because of its strong empirical stability. The objective function of PPO is: ; in, This represents the probability ratio between the current strategy and the reference strategy. For the estimated advantage, This is the clipping threshold.

[0041] In this embodiment, by defining GFR as the reward signal, the difference in generation accuracy between the generator using S-RAG document retrieval and simple top-k document retrieval is compared. This directly quantifies the value of retrieval quality in improving generation performance, solving the problem of insufficient robustness of traditional perfect match (EM) reward signals. Using generation accuracy (GenAcc) as the evaluation metric further improves the accuracy and reliability of the reward signal.

[0042] like Figure 1 The diagram shown is an overall flowchart of the retrieval enhancement model construction method, clearly presenting the complete process from model construction to evaluation: 1) Model building phase: It is clear that the retrieval model built in this embodiment consists of a trainable searcher LLM(S) and a frozen generator LLM(G), and the two are modularly separated.

[0043] 2) Initialization phase: Input the initial question Q, and retrieve the first k documents through retrieval. After filtering, the initial context is formed and passed to the generator to obtain the initial answer; 3) Multi-round search-selection loop phase: The search engine generates queries. Relevant documents were retrieved using the search engine SerpApi. Then filter the important document subset. Update the context and determine whether the loop should terminate based on whether the search needs to continue. 4) Training and Evaluation Phase: Calculate the GFR reward signal based on the generated results, update the searcher parameters through reinforcement learning using the PPO algorithm, and finally compare the trained S-RAG model (trained searcher + frozen generator) with the current RAG scheme on the test dataset to output the evaluation results.

[0044] This embodiment employs the PPO algorithm for reinforcement learning optimization. A probability ratio clipping mechanism ensures the stability of policy updates, avoiding training oscillations caused by excessively large update magnitudes. The complete search trajectory is included in the optimization scope, and the retrieval strategy is directly optimized in conjunction with the GFR reward signal. This allows the search engine to learn retrieval logic that better fits the generation requirements, significantly improving the relevance and effectiveness of retrieved documents. Compared to traditional reinforcement learning algorithms, the empirical stability advantage of PPO further enhances the efficiency and convergence reliability of model training.

[0045] This embodiment demonstrates the experimental results. The experimental design is as follows: Based on existing research, a training set was constructed by combining samples from Natural Question Answering (NQ) and HotpotQA. Due to the wide range of differences in the generation accuracy calculation... The system might incorrectly accept answers to questions with semantic negation, such as treating "not true" as a match for "true." Therefore, all yes / no and true / false questions were removed from the training set to ensure reliable reward signals. To focus training on more difficult examples, the training set was filtered out of samples where the frozen generator LLM (Qwen2.5-14B-Instruct) had already generated correct answers using Naive RAG retrieval, which significantly reduced the size of the training set. For the test dataset, five general-domain QA benchmark datasets were selected: NQ, TriviaQA, PopQA, HotpotQA, and 2wiki. The first three datasets are for single-hop reasoning (i.e., simple question answering), while the latter two datasets are for multi-hop reasoning complex question answering.

[0046] Additionally, the prompt words used by the searcher S, generator G, and correctness check (judge_check) are shown in Table 2: Table 2. Prompt words used by the search engine LLM (S) and the generator LLM (G).

[0047] In the preferred embodiment, the search engine LLM(S) and the generator LLM(G) interact using preset prompts, whereby the search engine's prompts guide it through the logical execution of query generation, document selection, and loop termination checks. The generator's prompts are used to guide users to answer questions directly based on the provided context, without including any extraneous text.

[0048] This embodiment clarifies the functional positioning of the searcher and generator by pre-setting dedicated prompts: the searcher prompts guide it through the core logic of query generation, document filtering, and iterative judgment, improving the targeting and standardization of the retrieval process; the generator prompts require it to answer questions directly based on context, avoiding redundant text and improving the simplicity and accuracy of answer generation; thereby reducing ambiguity in model interaction, improving the collaboration efficiency of the searcher and generator, and further enhancing the overall system's smoothness and output quality.

[0049] This embodiment compares S-RAG with a range of different RAG systems: (1) End-to-end fine-tuning. Fully fine-tuned models for joint retrieval and generation using outcome-based reinforcement learning or supervised learning: Search-R1(3B / 7B), SFT(3B), and R1(7B), where the models for SFT and R1 are both Qwen2.5-3 / 7B-Instruct.

[0050] (2) Static retrieval + frozen generator. Methods that use fixed or script strategies to retrieve documents and then pass them to the frozen generator: RAG-BM25, RAG-E5 (retrieve via BM25 or E5-base).

[0051] During S-RAG training, Qwen2.5-7B-Instruct was used as the base searcher LLM for training, and Qwen2.5-14B-Instruct was used as the freeze generator for answer generation and for correctness checking of reward calculation. The frozen generator was trained and evaluated on five NVIDIA A100 GPUs.

[0052] Analysis of experimental results: As shown in Table 3, the S-RAG retrieval model was evaluated on five general domain QA benchmarks. Generation accuracy is reported as the primary metric, and detailed comparisons of baselines, reward functions, and training efficiency are provided. The main result is generation accuracy, with exact match scores in parentheses. "Searcher" indicates the number of parameters in the searcher model. "Train" indicates the amount of training data used to train the searcher. DeepRetrievalBM25 was trained on NQ, as were Search-R1 and s3. Results are the average of three runs. Experimental results show that the S-RAG retrieval model outperforms the baseline on five general datasets with only 2.4k samples. Furthermore, the S-RAG retrieval model consistently outperforms Search-R1 in search quality, indicating that most of the performance improvement of RAG stems from enhanced search capabilities rather than alignment-based output generation.

[0053] Table 3 shows the test results of S-RAG and other methods on different datasets.

[0054] like Figure 2 The figure shows the reward curves for different initial document counts k (k=3, k=5, k=8). The horizontal axis represents the number of training steps (0-50 steps), and the vertical axis represents the reward value (0-0.6). The three curves correspond to the training reward trends for three initial document counts k=3, k=5, and k=8, respectively. During training, the rewards for all three k values ​​gradually increase with the number of steps, eventually converging, indicating that the model training is effective. The highest reward peak occurs when k=8, the peak value is close to that of k=8 when k=5, and the peak value is relatively low when k=3. This indicates that the initial number of retrieved documents has an impact on the reward increase. However, when k reaches 5, further increasing the value of k brings limited reward increase, which verifies the rationality of the k value range (3-8) and also reflects that the model can efficiently utilize the retrieved document information.

[0055] As shown in Table 4, an ablation study was conducted on the search loop component of S-RAG, specifically whether the first query of the original question was used as the training initialization and whether the selection scope in the loop was set to documents. The data in Table 3 reports the generation accuracy under different ablation conditions. The results show that removing the search initialization leads to a significant performance decrease, and removing "document selection" sometimes brings better performance but significantly increases the use of input tokens, thus reducing overall efficiency. Through a multi-round search-selection loop mechanism, a balance between retrieval accuracy and efficiency is achieved. While filtering key information and ensuring performance, the number of input tokens is reduced, improving the overall efficiency of the system.

[0056] Table 4. S-RAG multi-round search-selection of ablation experiments for cyclic components.

[0057] Table 5 compares the training time of Search-R1 and S-RAG when tested on five NVIDIA A100 GPUs with a batch size of 120. Because S-RAG requires frozen LLM for reward calculation during training, its single-step speed is slower. However, training S-RAG requires only 20 PPO steps (2.4k examples), while Search-R1 requires 2,100 steps (170k examples). Even considering the higher per-step cost due to LLM-based reward calculation, the total clock time is reduced by approximately 33 times. Furthermore, S-RAG avoids retrieval pre-training and uses a smaller 7B policy model, which also reduces the RL training cost of the RAG model.

[0058] Table 5. Training time on five NVIDIA A100 GPUs with a batch size of 120.

[0059] Experimental results show that on three general datasets, the S-RAG framework successfully outperformed the baseline model using only 2.4k training samples, significantly improving the performance of open-domain question answering systems. In terms of training efficiency, unlike the EM method which requires approximately 2.1k training steps, S-RAG utilizes a frozen generator to perform reinforcement learning training only on the search engine, converging in approximately 20 PPO steps. Although each training step is costly due to the need to call a large language model to calculate rewards, its extremely fast convergence speed completely offsets this overhead, resulting in a total training time that is approximately 33 times shorter than Search-R1, achieving a significant reduction in total cost. Furthermore, the reward curve of S-RAG shows that the model can quickly learn the most useful documents, with performance approaching peak value (k=8) even with a small retrieval range (k=5), indicating excellent sample utilization and accurate retrieval judgment, avoiding unnecessary computational waste. Furthermore, ablation studies reveal the importance of its component design: using the original problem for search initialization is key to maintaining high performance, while the document selection mechanism improves the overall operating efficiency of the system by significantly reducing the number of input tokens without significantly sacrificing performance.

[0060] In summary, S-RAG optimizes the core retrieval step of the RAG process, achieving superior performance with less data, less time, and lower cost through a highly intelligent and resource-efficient approach. This provides a promising solution for building efficient and practical next-generation question-answering systems.

[0061] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A method for constructing a retrieval enhancement model based on PPO, characterized in that, The search engine and generator are decoupled, and the search engine is trained using reinforcement learning while keeping the generator frozen. This includes the following steps: S1: Construct an S-RAG retrieval model consisting of a search engine LLM(S) and a generator LLM(G), wherein the search engine LLM(S) is modularly separated from the generator, and the generator LLM(G) remains frozen during training. S2: Use the search engine SerpApi to retrieve the first k relevant documents D0 based on the initial question Q, form the initial context, and then pass it into the generator G to obtain the initial answer; S3: The problem of rewriting the query based on Q using the search engine LLM(S) ; S4: Search engine SerpAPI based on The search retrieved the top k relevant documents. ; S5: Searcher S retrieves documents from a new round. Select important documents to form context And determine whether enough information has been found to stop the loop; Steps 3 through 5 are performed alternately to form a multi-round search-selection loop; S6: Combine the initial problem Q with the important documents obtained from the multi-round search-selection loop. The context is provided to the freeze generator LLM(G), which generates the output and calculates the retrieved enhancement gain GFR based on the returned result; S7: Implement reinforcement learning of the searcher LLM(S) based on the Proximal Policy Optimization (PPO) algorithm, where the reward signal is GFR; S8: Test the S-RAG retrieval model, which consists of the trained search engine LLM(S) and the frozen generator LLM(G), on another dataset and compare it with the current RAG retrieval scheme.

2. The method for constructing a retrieval enhancement model based on PPO according to claim 1, characterized in that, Before performing the multi-round search-selection loop, the following is included: Initialization operation: Given a problem Q, use For the initial query, the first k documents to be tested are retrieved. Then select a subset This constitutes the initial context.

3. The method for constructing a retrieval enhancement model based on PPO according to claim 2, characterized in that, The multi-round search-selection loop includes the following sub-steps: In each iteration of the loop, the following steps are executed sequentially: The search engine generates a query and return the results in a preset query format; According to the query Search related documents And return the results in a preset information format; Based on the prompts, retrieve documents Select a useful subset of relevant documents And return the results in a preset selection format; When no new documents are updated or the round limit is reached, output a declaration message in a preset end format and the loop ends; After the loop terminates, the final context is obtained and passed to the frozen generator to generate the final output.

4. The method for constructing a retrieval enhancement model based on PPO according to claim 1, characterized in that, In step S3, the definition of the search enhancement gain GFR is: ； in, As the standard answer, Retrieved from the original question One result, For a task-specific metric, GenAcc (Generated Accuracy) is used, expressed as: ； In the formula, For rapid span matching testing, For correctness checks based on lightweight LLM.

5. The method for constructing a retrieval enhancement model based on PPO according to claim 4, characterized in that, Before training the search engine, a sample selection step is also included: Selecting training samples: Pre-calculate the baseline accuracy term Acc(G(Q,D_RAG),A), filter out samples with a baseline accuracy term equal to 0, and only keep queries that are difficult to solve with simple RAG as training samples.

6. The method for constructing a retrieval enhancement model based on PPO according to claim 4, characterized in that, The reinforcement learning employs a proximal policy optimization PPO algorithm, including: Using GFR rewards, the searcher's search strategy is optimized through reinforcement learning, with each deployment including a complete search trajectory: the issued query, document selection, and stopping decision; once the final context is built... generator This will generate the answer and calculate the GFR reward, and the generator will be optimized using the PPO algorithm; The objective function of PPO is: ； in, This represents the probability ratio between the current strategy and the reference strategy. For the estimated advantage, This is the clipping threshold.

7. The method for constructing a retrieval enhancement model based on PPO according to claim 1, characterized in that, During the reinforcement learning training process, the generator remains frozen, and the gradient is backpropagated only through the searcher's strategy.

8. The method for constructing a retrieval enhancement model based on PPO according to claim 4, characterized in that, In S1, the training of the searcher and frozen generator models also includes: Qwen2.5-7B-Instruct was selected as the base searcher LLM for training; Qwen2.5-14B-Instruct was selected as the frozen generator for answer generation and the frozen scorer for judge_check; A training set was constructed by combining samples from Natural Question Answering (NQ) and HotpotQA. All yes / no and true / false questions were removed from the training set. NQ, TriviaQA, PopQA, HotpotQA, and 2wiki were selected as test datasets. Among them, NQ, TriviaQA, and PopQA are single-hop inference datasets, while HotpotQA and 2wiki are multi-hop inference datasets. Training parameter settings: During training, the number of training samples was set to 2.4k, the batch size to 120, and the number of training steps to 20 PPO steps.

9. The method for constructing a retrieval enhancement model based on PPO according to claim 3, characterized in that, The scoring criteria for document selection are determined based on the relevance of the document to the query, the completeness of the content, and its contribution to generating the correct answer. The document selection follows the following rules: newly added documents are marked as important, documents without important tags are deduplicated, and a new round of document screening is obtained by combining the two. The number of important documents selected each time does not exceed 3.

10. The method for constructing a retrieval enhancement model based on PPO according to claim 1, characterized in that, The search engine LLM(S) and the generator LLM(G) interact using preset prompts, wherein the prompts of the search engine are used to guide it to complete the logical execution of query generation, document selection and loop termination judgment; The generator's prompts are used to guide users to answer questions directly based on the provided context, without including any extraneous text.

Images