Multi-modal retrieval enhancement generation method based on gradual group relative strategy optimization

Abstract

The invention belongs to the technical field of artificial intelligence and multi-modal large model reasoning enhancement, and discloses a multi-modal retrieval enhancement generation method based on gradual group relative strategy optimization. A step-by-step reasoning track construction mechanism is introduced, an original problem is disassembled into a plurality of sub-problems, and in each step, a new retrieval query is autonomously generated in combination with reasoning history and current information requirements, and a most appropriate knowledge source is selected for evidence retrieval; and in the reasoning process, each step of decision and answer obtains a fine-grained reward signal. According to the method, a group relative strategy optimization method is adopted, the query quality of each reasoning step, the knowledge base routing accuracy, the answer content format compliance and the final answer accuracy are used as step-by-step rewards for joint modeling, and model parameters are optimized through global and local multiple feedback signals. The method is remarkably superior to the existing similar technology in tasks such as multi-class multi-modal open domain question answering and complex reasoning, and has excellent answer accuracy, retrieval efficiency and multi-modal adaptive capacity.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and multimodal large model reasoning enhancement technology, and in particular to a multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization, which is suitable for scenarios such as intelligent question answering, multimodal reasoning, and complex information extraction. Background Art

[0002] With the development of artificial intelligence, multimodal retrieval-enhanced generation methods are becoming a mainstream approach to improving the reasoning and generation capabilities of large models. Existing multimodal retrieval-enhanced generation methods typically incorporate information from heterogeneous knowledge bases such as text, images, and tables to assist in large model reasoning and generation, significantly mitigating errors and hallucinations that can occur in closed knowledge contexts. Despite recent advances in modular retrieval and static planning, significant limitations persist in practical applications. On the one hand, existing technologies often rely on fixed retrieval sequences and static processes, making it difficult to implement dynamic information exchange during reasoning and unable to adjust retrieval paths in real time based on problem decomposition and contextual requirements. This results in insufficient performance for complex multi-hop, multimodal problems. On the other hand, existing methods rarely fully utilize the inherent step-by-step reasoning and dynamic planning capabilities of large models. Training signals often focus solely on the final answer, lacking fine-grained supervision across multiple stages, including intermediate decision-making, querying, and routing. Consequently, existing multimodal retrieval-enhanced generation methods struggle to meet the complex scenarios required by real-world engineering applications, whether in terms of reasoning depth, generalization performance, or retrieval efficiency.

[0003] "Yu X, Yang Z, Chen C. Unveiling the Potential of Multimodal RetrievalAugmented Generation with Planning[J].arXiv preprint arXiv:2501.15470,2025." focuses on the reasoning planning task of multimodal retrieval augmented generation, optimizing the retrieval and reasoning processes of large models by introducing multi-step reasoning chains and planning agents. CogPlanner emphasizes the model's ability to autonomously plan each step of query, evidence acquisition, and reasoning. It also introduces a dedicated benchmark to evaluate chain planning and answer integration capabilities in multimodal reasoning. Although CogPlanner improves the performance of large models in complex reasoning processes, its reward signal is primarily focused on the final answer, with weak dynamic feedback from intermediate steps of reasoning. This leaves room for improvement in the model's dynamic planning and routing.

[0004] "Li Y, Li Y, Wang X, et al. Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self-adaptive planning agent[J].arXiv preprint arXiv:2411.02937,2024." is a retrieval-augmented generation method for complex multimodal reasoning tasks. Its key feature is the introduction of an adaptive planning mechanism that dynamically decomposes complex problems into a series of subproblem chains based on the current reasoning process and retrieved content. For each subproblem, OmniSearch selects the most appropriate knowledge base for retrieval in real time, including data sources in different modalities such as text and images, effectively simulating the step-by-step verification and reasoning behavior of human experts. This method improves information acquisition efficiency when solving complex problems, but its planning and routing mechanisms rely heavily on manual rules and prompt patterns, lack fine-grained strategy optimization for intermediate decision-making, and have certain limitations in generalization.

[0005] In "Yeo W, Kim K, Jeong S, et al. UniversalRAG: Retrieval-Augmented Generation over Multiple Corpora with Diverse Modalities and Granularities[J].arXiv preprint arXiv:2504.20734,2025.", a unified retrieval-augmented generation framework that supports multimodal, granular, and multi-type knowledge sources is proposed. UniversalRAG can retrieve relevant knowledge from a variety of heterogeneous sources, including structured, semi-structured, and unstructured, and organically integrate this information into the generation and inference process of large models, improving the model's adaptability to diverse query scenarios. This approach addresses the problem of a single knowledge source being unable to meet the requirements of complex multimodal reasoning. However, it still lacks more sophisticated policy training and reward feedback in areas such as dynamic routing and multi-step optimization of the reasoning chain, resulting in limited active adaptability of the model in multi-step reasoning and cross-modal planning. Summary of the Invention

[0006] The technical problem addressed by this invention is that existing generative models for multimodal retrieval enhancement suffer from low retrieval efficiency, insufficient dynamic adaptability, and limited generalization performance in complex multimodal problem scenarios. To address these technical challenges, this invention proposes a generative method for multimodal retrieval enhancement based on step-by-step group relative strategy optimization.

[0007] The technical solution of the present invention is a multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization, which specifically includes the following steps:

[0008] Step 1: Categorize and organize data containing multiple modalities, implement unified numbering and indexing, and build an extensible hybrid knowledge base D for subsequent retrieval;

[0009] Step 2: Generate step-by-step reasoning trajectories based on the prompted large reasoning model or multimodal large language model, and only retain the reasoning endpoint a by rejecting sampling n+1 The reasoning trajectory that is completely consistent with the true answer is taken as the true reasoning trajectory; the initial query and the corresponding true reasoning trajectory are combined to form a true reasoning trajectory dataset;

[0010] Step 3: Based on the real reasoning trajectory dataset constructed in Step 2, a stepwise group relative strategy optimization method is used to define stepwise rewards and optimize the reasoning and retrieval decision process of the multimodal large language model.

[0011] Step 4: Based on the multimodal large language model optimized in step 3, perform step-by-step reasoning on the input question, retrieve information, and finally generate an answer.

[0012] The construction of the step-by-step reasoning trajectory is specifically as follows:

[0013] Generate step-by-step reasoning trajectories by prompting a large reasoning model or a multimodal large language model; each step-by-step reasoning trajectory R contains multiple reasoning steps, each of which is represented by (q i ,s i ,d i ,a i ) four-tuple composition: where q i The subquery generated in step i, s i is the selected retriever, d i is the evidence fragment retrieved, a i Based on d i The intermediate answer generated with the current context.

[0014] The reasoning and retrieval decision process of the multimodal large language model is as follows:

[0015] The multimodal large language model first gradually infers the trajectory R based on the initial question q0 and the acquired history 1:i-1 , generate the thinking content of the current step t i ;

[0016] Then according to t i , q0 and historical step-by-step reasoning trajectory R 1:i-1 , automatically generate this step intermediate query q i With selected retrievers i ;

[0017] According to s i , call the corresponding retrieval module from the hybrid knowledge base D, using q i Retrieve the most relevant evidence fragments i , d i =Search(q i ,D(s i ));

[0018] q i d i , t i The three are input into the multimodal large language model to generate the answer, and the intermediate answer a is obtained. i ;

[0019] The middle answer a i With q i d i , t i Merge to form R i , and the above historical reasoning trajectory R 1:i-1 Merge to form the current historical step-by-step reasoning trajectory R 1:i , and judge whether the termination condition is met; if not, proceed to the next iteration;

[0020] After the termination condition is met, the multimodal large language model is based on the initial question q0 and the complete historical trajectory R 1:n Generate the final answer a n+1 .

[0021] The step-by-step reward is divided into calculating the reward value for generating an intermediate query at each step of the multimodal large language model and the reward value after the query is completed, specifically including:

[0022] Define query reward r ask (q i ), used to measure the semantic similarity between the generated intermediate query and the real intermediate query;

[0023] Define routing reward r route (s i ), a retriever that determines the selection of a multimodal large language model i The retriever selected from the ground-truth trajectory Whether it is consistent, accurate is 1, wrong is 0;

[0024] Define format reward r format , assess whether the output of the multimodal large language model strictly follows the preset tagging specifications. If the format is correct, the reward is 1, otherwise 0;

[0025] Define the answer reward r answer (a i), the intermediate answers are evaluated using the F1-Recall indicator, and the final answers are evaluated using accuracy;

[0026] The weighted combination of multiple rewards serves as the total reward signal for this step, which is used to design the loss function for subsequent step-by-step group relative strategy optimization.

[0027] The step 3 is specifically as follows:

[0028] Step 3.1: Use the frozen multimodal large language model from the previous training round Sample the next prediction step based on each inference step in the true inference trajectory;

[0029] Step 3.2: Define the normalized odds estimate score Calculated by normalizing the mean and standard deviation of the reward, where k represents the kth sample output and t represents the tth token in the output;

[0030] Calculate the GRPO loss L for each sample trajectory GRPO (x, r), and a stepwise group relative policy optimization method is used to perform gradient updates to optimize the model parameters, where x is the input context information and r is the corresponding stepwise reward.

[0031] The step-by-step group relative strategy optimization method is specifically as follows:

[0032] Based on the above real reasoning trajectory data and the total reward signal of each step, the step-by-step group relative strategy optimization loss function L is defined:

[0033]

[0034] r (1) =r format (q i ,s i )×(αr ask (q i )+βr route (s i ))

[0035] r (2) =r format (a i )×r answer (a i )

[0036] L GRPO is the loss function for single-step group relative strategy optimization, q0 is the initial problem, R is the real reasoning trajectory, r (1) Reward function for generating and selecting retrievers for intermediate queries, r (2) is the reward function for generating the answer, q iis the intermediate query in the actual reasoning step, d i is the document retrieved in the actual inference step; α and β are hyperparameters that balance the importance of query relevance and correct routing;

[0037] For each reasoning step i under the step-by-step reasoning trajectory, a group sampling strategy is adopted, based on the distribution ratio of the multimodal large language model output of the current round of training and the multimodal large language model output of the previous training round, combined with the advantage estimation score After normalization, the loss function of single-step group relative strategy optimization is constructed; L GRPO (x,r)

[0038]

[0039] G is the number of group samples; ∈ is the truncation hyperparameter; clip represents the truncation function; O k Refers to the kth prediction step of the sampling; |O k | represents O k The embedding vector of k |tokens; Indicates that the currently trained multimodal large language model outputs o based on input x and t-1 token outputs k,<t The probability of o is higher than that of the multimodal large language model output in the previous training round. k,<t Ratio of probabilities;

[0040] For a given input x, extract a set of prediction steps corresponding to a certain inference step {O1, O2, ..., O G}, and through the reward function r (1) or r (2) Get reward {r1,r2,…,r G Normalized advantage estimate score for each token The calculation is as follows:

[0041]

[0042] mean is the mean, std is the standard deviation;

[0043] The parameters of the multimodal large language model are gradually optimized through a step-by-step group relative strategy optimization method.

[0044] The termination condition is: when the preset maximum number of inference steps is reached, or when the multimodal large language model determines that sufficient information has been obtained to generate an accurate final answer, the inference loop is terminated.

[0045] The multimodal large language model simultaneously supports the understanding and generation tasks of text, images, and tables, and the step-by-step reasoning optimization, retrieval routing, and answer generation are all completed under the same multimodal large language model architecture.

[0046] Beneficial effects of the present invention: The method proposed in the present invention can give large models the ability to automatically plan reasoning links, dynamically route cross-modal knowledge bases, and accumulate evidence in multiple steps, thereby achieving efficient reasoning and generation of complex problems. The method introduces a step-by-step reasoning trajectory construction mechanism, allowing the model to decompose the original problem into several sub-problems. At each step, it can autonomously generate new retrieval queries and select the most appropriate knowledge source for evidence retrieval based on the reasoning history and current information needs; the decision and answer of each step in the reasoning process can obtain fine-grained reward signals. Furthermore, the present invention adopts a group relative strategy optimization method, and jointly models the query quality of each reasoning step, the accuracy of the knowledge base routing, the format compliance of the answer content, and the accuracy of the final answer as step-by-step rewards, and optimizes the model parameters through global and local multiple feedback signals. Experimental results show that this method significantly outperforms existing similar technologies in tasks such as multi-class multimodal open domain question answering and complex reasoning, and performs well in terms of answer accuracy, retrieval efficiency, and multimodal adaptability, and has broad theoretical and engineering value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the effect of the multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization;

[0048] Figure 2 A schematic diagram of the basic model effect;

[0049] Figure 3 Schematic diagram of the inference model. DETAILED DESCRIPTION

[0050] Step 1: Build a multimodal hybrid knowledge base containing multiple modal data such as text, images, and tables;

[0051] Step 1.1: Collect multimodal raw data, including large-scale text, images, tables, and other different types of data resources to ensure that the knowledge base can support multiple reasoning tasks.

[0052] Step 1.2: Categorize and organize the raw data, store text, images, tables and other data in the corresponding database, implement unified numbering and indexing, and build an extensible hybrid knowledge base D.

[0053] Step 1.3: Design a dedicated retrieval interface for each type of data to implement efficient vector retrieval or content-based search for text, images, and tables, ensuring that evidence from each modality can be retrieved and integrated in a timely manner in subsequent reasoning.

[0054] Step 2: Construct step-by-step reasoning data. Generate step-by-step reasoning trajectories based on a large reasoning model or a multimodal large language model, and filter them to form a high-quality dataset of real reasoning trajectories.

[0055] Step 2.1: Design a multi-step reasoning task, input the original problem q0, and gradually decompose the complex problem through manual or rule-based methods to obtain a multi-step chain of sub-problems, covering cross-modal and multi-hop information reasoning scenarios.

[0056] Step 2.2: Use the large model to automatically generate multi-step reasoning trajectories. Each reasoning trajectory R contains multiple reasoning steps. Each reasoning step is composed of (q i ,s i ,d i ,a i ) four-tuple composition: where q i The subquery generated in step i, s i is the selected retriever, d i is the evidence fragment retrieved, a i Based on d i The intermediate answer generated with the current context.

[0057] Step 2.3: By rejection sampling, only those inference endpoints a are retained n+1 The trajectories that are completely consistent with the standard answers eventually form a high-quality real reasoning trajectory dataset (q0, R) as the basis for subsequent strategy optimization training.

[0058] Step 3: Based on the real reasoning trajectory dataset constructed in Step 2, a stepwise group relative strategy optimization method is used to define stepwise rewards and optimize the reasoning and retrieval decision process of the multimodal large language model.

[0059] Furthermore, a step-by-step group relative strategy optimization algorithm is adopted to define step-by-step rewards, optimize the reasoning and retrieval decision process of the multimodal large language model, and improve the model's reasoning accuracy and generalization ability.

[0060] The multimodal large language model is used to generate an intermediate query based on the current reasoning state, and the most appropriate retriever is selected to route to the corresponding knowledge base for retrieval to obtain the evidence most relevant to the query; an intermediate answer is generated based on the current context information and the retrieved evidence, and the process is executed in a loop until the termination condition is met; in the final reasoning step, the initial query and the intermediate answers of the entire reasoning process are used as input to generate the final answer;

[0061] In actual reasoning, the multimodal large language model first gradually reasons the trajectory R based on the current question q0 and the acquired history. 1:i-1 , generate the thinking content of the current step t i , t i=f think (q0,R 1:i-1 ), where f think is the inference function.

[0062] According to t i , q0 and historical step-by-step reasoning trajectory R 1:i-1 , automatically generate this step intermediate query q i With selected retrievers i , (q i ,s i )=f ask (q0,R 1:i-1 ,t i ).

[0063] According to s i , call the corresponding retrieval module from the hybrid knowledge base D, using q i Retrieve the most relevant evidence fragments i , d i =Search(q i ,D(s i )).

[0064] q i d i , t i The three inputs generate the function and get the intermediate answer a i , a i =f answer (t i ,q i ,d i ).

[0065] Append the result of this step to the historical trajectory R 1:i , and judge whether the termination condition is met; if not, enter the next iteration.

[0066] Each reasoning trajectory ends when it reaches the maximum number of steps n or the model determines that it has obtained sufficient information, and is finally represented by (q0, R 1:n ) and the final thinking result t n+1 Generate the final answer a n+1 =f final (q0,R 1:n ,t n+1 ).

[0067] After the final answer is output, the multimodal large language model records the complete reasoning trajectory, including all sub-queries, routes, retrieval evidence, and intermediate and final answers, providing a fine-grained data foundation for subsequent supervision and strategy optimization.

[0068] The specific progressive rewards are:

[0069] Build a reward system that incorporates intermediate query generation, knowledge base routing selection, format specifications, intermediate answers, and final answer quality at each step in the reasoning chain into reward assessment.

[0070] Query Rewards ask (q i ): measure the q generated by the model i With real subquery The semantic similarity of , such as calculated by vector dot product.

[0071] Routing reward r route (s i ): Knowledge base s for judging the selection of multimodal large language models i With the real trajectory Whether it is consistent, 1 is accurate and 0 is wrong.

[0072] Format Rewards format :Assess whether the output of the multimodal large language model strictly follows the preset tag specification. If the format is correct, the reward is 1, otherwise 0. The reasoning process, intermediate query and called retriever should be " <think> ...< / think> "," <ask> ...< / ask> "and" <ret> ...< / ret> When generating intermediate and final answers, we define the correct format of the reasoning process and answer to be surrounded by “ <think> ...< / think> "and" <answer> ...< / answer> ” mark.

[0073] Answer reward r answer (a i ): The intermediate answers are evaluated using the F1-Recall indicator, and the final answer is evaluated using the accuracy rate.

[0074] The intermediate answer reward is expressed as:

[0075]

[0076] in represents the true intermediate answer of step i.

[0077] The F1-Recall calculation formula is:

[0078]

[0079] in Indicates the predicted answer a i With the real answer The reward for the final answer is calculated using accuracy:

[0080]

[0081] in The true answer indicating the final answer.

[0082] The weighted combination of multiple rewards serves as the total reward signal for this step and participates in the design of the subsequent strategy optimization objective function.

[0083] Using the above real reasoning trajectory data and the reward signal at each step, we set the step-by-step group relative strategy optimization objective function L:

[0084]

[0085] Among them, L GRPO is the single-step group relative strategy optimization loss, q0 is the original problem, R is the real reasoning trajectory, r (1) 、r (2) For gradual reward; (1) Reward function for generating and selecting retrievers for intermediate queries, r (2) is the reward function for generating the answer, q i ,d i are the query and retrieval evidence in the real reasoning step, respectively.

[0086] For each inference step i, a group sampling strategy is adopted, and the output of the currently trained multimodal large language model is based on the input x and t-1 token output o k,<t The probability of o is higher than that of the multimodal large language model output in the previous training round. k,<t The ratio of probabilities, combined with rewards After normalization, the loss function for single-step group relative strategy optimization is constructed.

[0087]

[0088] G is the number of group samples.

[0089] By optimizing the parameters of the multimodal large language model through gradient descent, its reasoning trajectory generation, dynamic routing decision-making, and answer output can all reach global optimality in complex multimodal scenarios.

[0090] To further demonstrate the effectiveness of the proposed method, we selected a case study from the visual QA task of the InfoSeek dataset. Figure 1 As shown in Figure 3, we show the step-by-step reasoning process of R1-Router for a query asking for the nearest parent category of the bird shown in the image.

[0091] In the first reasoning step, our method initiates a reflection process and decomposes the task into a sub-goal: performing visual analysis. It formulates an intermediate query: "What is the name of the bird shown in the image?" and selects a text-image retriever to obtain relevant visual information. Using this retriever, our method retrieves semantically similar image descriptions from the knowledge base, containing key entities such as "Bay-breasted Warbler" that describe the bird in the given image. Notably, our method performs a step of reflection, verifying that the bird is indeed a Bay-breasted Warbler based on visual features mentioned in the retrieved evidence (e.g., breast pattern). This demonstrates the effectiveness of R1-style deep reasoning in RAG-based modeling. In the second step, our method formulates a follow-up query: "What is the nearest parent category of Bay-breasted Warbler?" and selects a text retriever to gather factual knowledge. Using this retrieved content, the R1-Router confirms the genus of the Bay-breasted Warbler as "Setophaga" and ultimately outputs the correct final answer. This case study demonstrates R1-Router's ability to dynamically decide when and from which knowledge sources to retrieve to solve a problem, showcasing its powerful reasoning and retrieval planning capabilities.

[0092] Tables 1 and 2 compare the effects of the present invention and existing methods in multiple modal open and question-answering tasks within and outside the distribution, where the data values ​​represent the answer recall rate of the model on the relevant reasoning data. It can be seen that the present invention has achieved the best results. In the knowledge-intensive task scenario, the accuracy effect of the present invention not only surpasses the traditional retrieval enhancement method, such as the method of using a retriever to perform single or multiple iterative retrieval on a fixed knowledge base, but also outperforms the method specifically used for dynamically selecting retrieval information from multiple knowledge bases, including OmniSearch, CogPlanner and UniversalRAG methods, which fully demonstrates the significant advantages of the present invention in multimodal retrieval enhancement.

[0093] Table 1 Main experimental results within the distribution

[0094]

[0095] Table 2 Main experimental results outside the distribution

[0096]

Claims

1. A multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization, characterized in that: The specific steps are as follows: Step 1: Categorize and organize data containing multiple modalities, implement unified numbering and indexing, and build an extensible hybrid knowledge base D for subsequent retrieval; Step 2: Generate step-by-step reasoning trajectories based on the prompted large inference model or multimodal large language model, and only retain the inference endpoint a by rejection sampling n+1 The reasoning trajectory that is completely consistent with the true answer is regarded as the true reasoning trajectory; Combine the initial query with the corresponding true reasoning trajectory to form a true reasoning trajectory dataset; Step 3: Based on the real reasoning trajectory dataset constructed in Step 2, a stepwise group relative strategy optimization method is used to define stepwise rewards and optimize the reasoning and retrieval decision process of the multimodal large language model. Step 4: Based on the multimodal large language model optimized in step 3, perform step-by-step reasoning on the input question, retrieve information, and finally generate an answer.

2. The multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization according to claim 1 is characterized in that: The construction of the step-by-step reasoning trajectory is specifically as follows: Generate step-by-step reasoning trajectories by prompting a large reasoning model or a multimodal large language model; each step-by-step reasoning trajectory R contains multiple reasoning steps, each of which is represented by (q i ,s i ,d i ,a i ) four-tuple composition: where q i The subquery generated in step i, s i is the selected retriever, d i is the evidence fragment retrieved, a i Based on d i The intermediate answer generated with the current context.

3. The multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization according to claim 2 is characterized in that: The reasoning and retrieval decision process of the multimodal large language model is as follows: The multimodal large language model first gradually infers the trajectory R based on the initial question q0 and the acquired history 1:i-1 , generate the thinking content of the current step t i ; Then according to t i , q0 and historical step-by-step reasoning trajectory R 1:i-1 , automatically generate this step intermediate query q i With selected retrievers i ; According to s i , call the corresponding retrieval module from the hybrid knowledge base D, using q i Retrieve the most relevant evidence fragments i , d i =Search(q i ,D(s i )); q i d i , t i The three are input into the multimodal large language model to generate the answer, and the intermediate answer a is obtained. i ; The middle answer a i With q i d i , t i Merge to form R i , and the above historical reasoning trajectory R 1:i-1 Merge to form the current historical step-by-step reasoning trajectory R 1:i , and judge whether the termination condition is met; If not satisfied, proceed to the next iteration; After the termination condition is met, the multimodal large language model is based on the initial question q0 and the complete historical trajectory R 1:n Generate the final answer a n+1 .

4. The multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization according to claim 1 is characterized in that: The step-by-step reward is divided into calculating the reward value for generating an intermediate query at each step of the multimodal large language model and the reward value after the query is completed, specifically including: Define query reward r ask (q i ), used to measure the semantic similarity between the generated intermediate query and the real intermediate query; Define routing reward r route (s i ), a retriever that determines the selection of a multimodal large language model i The retriever selected from the ground-truth trajectory Whether it is consistent, accurate is 1, wrong is 0; Define format reward r format , assess whether the output of the multimodal large language model strictly follows the preset tagging specifications. If the format is correct, the reward is 1, otherwise 0; Define the answer reward r answer (a i ), the intermediate answers are evaluated using the F1-Recall indicator, and the final answers are evaluated using accuracy; The weighted combination of multiple rewards serves as the total reward signal for this step, which is used to design the loss function for subsequent step-by-step group relative strategy optimization.

5. The multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization according to claim 1 is characterized in that: The step 3 is specifically as follows: Step 3.1: Use the frozen multimodal large language model from the previous training round Sample the next prediction step based on each inference step in the true inference trajectory; Step 3.2: Define the normalized odds estimate score Calculated by normalizing the mean and standard deviation of the reward, where k represents the kth sample output and t represents the tth token in the output; Calculate the GRPO loss L for each sample trajectory GRPO (x, r), a stepwise group relative policy optimization method is used to perform gradient updates to optimize the model parameters, where x is the input context information and r is the corresponding stepwise reward.

6. The multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization according to claim 1 or 5, characterized in that: The step-by-step group relative strategy optimization method is specifically as follows: Based on the above real reasoning trajectory data and the total reward signal of each step, the step-by-step group relative strategy optimization loss function L is defined: r (1) =r format (q i ,s i )×(αr ask (q i )+βr route (s i )) r (2) =r format (a i )×r answer (a i ) L GRPO is the loss function for single-step group relative strategy optimization, q0 is the initial problem, R is the real reasoning trajectory, r (1) Reward function for generating and selecting retrievers for intermediate queries, r (2) is the reward function for generating the answer, q i is the intermediate query in the actual reasoning step, d i is the document retrieved in the actual inference step; α and β are hyperparameters that balance the importance of query relevance and correct routing; For each reasoning step i under the step-by-step reasoning trajectory, a group sampling strategy is adopted, based on the distribution ratio of the multimodal large language model output of the current round of training and the multimodal large language model output of the previous training round, combined with the advantage estimation score After normalization, construct the loss function for single-step group relative strategy optimization; G is the number of group samples; ∈ is the truncation hyperparameter; clip represents the truncation function; O k Refers to the sampled k-th prediction step embedding vector, including |O k |tokens; Indicates that the currently trained multimodal large language model outputs o based on input x and t-1 token outputs k,<t The probability of o is higher than that of the multimodal large language model output in the previous training round. k,<t The ratio of probabilities; For a given input x, extract a set of embedding vectors {O1, O2, ..., O G }, and through the reward function r (1) or r (2) Get reward {r1,r2,…,r G Normalized advantage estimate score for each token The calculation is as follows: mean is the mean, std is the standard deviation; The parameters of the multimodal large language model are gradually optimized through a step-by-step group relative strategy optimization method.

7. The multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization according to claim 1 is characterized in that: The termination condition is: when the preset maximum number of inference steps is reached, or when the multimodal large language model determines that sufficient information has been obtained to generate an accurate final answer, the inference loop is terminated.

8. The multimodal retrieval enhancement generation method based on step-by-step group relative strategy optimization according to any one of claims 1 to 7, characterized in that: The multimodal large language model simultaneously supports the understanding and generation tasks of text, images, and tables, and the step-by-step reasoning optimization, retrieval routing, and answer generation are all completed under the same multimodal large language model architecture.

Images