A Retrieval Enhancement Generation Method and System Based on Multimodal Fusion and Reinforcement Learning Collaboration

Abstract

This invention relates to the field of information retrieval technology and discloses a retrieval enhancement generation method and system that combines multimodal fusion and reinforcement learning. The method receives a user-inputted original query and combines it with the multimodal context of the current iteration step to generate a subquery based on a large language model. The subquery and multimodal context are combined to form the current state, and the retrieval enhancement generation task is modeled as a Markov decision process. The large language model adaptively selects the optimal action from a predefined set of actions in the current state according to the decision strategy. Based on the optimal action, the corresponding multimodal retrieval operation is executed, and the acquired multimodal information is fused to generate intermediate or final answers to the subqueries. The intermediate answers are used to update the multimodal context. The large language model is offline trained and optimized through imitation learning and a calibration chain, and the fine-tuned model is used online to infer decision strategies and subqueries. This invention achieves more efficient and accurate processing of complex queries.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, specifically to a retrieval enhancement generation method and system that combines multimodal fusion and reinforcement learning. Background Technology

[0002] In recent years, Retrieval-Augmented Generation (RAG) technology has become an important means to improve the ability of large language models (LLMs) to handle knowledge-intensive tasks. Its development is mainly reflected in the following aspects: (1) RAG based on static text databases: represented by DPR and REALM, it enhances the input of LLM through a single dense text retrieval. It mainly relies on static knowledge sources such as Wikipedia, which lays the foundation for RAG technology, but it has limitations in handling complex queries and dynamic knowledge. (2) Evolution of iterative and proactive text retrieval RAG: represented by DeepRAG, FLARE, and Self-RAG, it introduces an iterative "think-retrieve-reason" cycle or a mechanism for LLM to actively trigger retrieval, which improves the flexibility and reasoning depth in handling complex text queries, but it is still mainly limited to text modality. (3) Preliminary exploration of visual information processing and application of reinforcement learning: Works represented by ColPali began to focus on efficient retrieval directly from images on document pages, solving the bottleneck of traditional OCR; at the same time, works represented by RAG-RL attempted to introduce reinforcement learning into text RAG optimization, showing the potential to optimize complex decision sequences, but multimodal fusion and intelligent decision-making are still immature.

[0003] Despite significant progress in improving the performance of large language models, existing RAG systems still exhibit numerous technical bottlenecks and limitations when facing increasingly complex real-world application scenarios, particularly in processing queries containing multiple modalities such as text and images. These limitations hinder the further realization of their potential, specifically in the following three aspects:

[0004] 1. Insufficient Multimodal Information Processing and Fusion Capabilities: Mainstream RAG systems primarily rely on the retrieval and processing of textual information. However, real-world documents (such as PDF reports, web pages, and presentations) often contain numerous visual elements such as charts, flowcharts, photos, and complex layouts. These elements carry rich, and sometimes irreplaceable, key information. Existing text-based RAG systems cannot directly and effectively extract and understand knowledge from these visual elements.

[0005] 2. Unimodality and Limitations of Decision-Making Mechanisms in Iterative Reasoning: The decision-making modules of existing iterative RAG systems primarily revolve around "whether to retrieve text" or "depending on parametric knowledge." In multimodal scenarios, the system lacks the intelligent decision-making ability to dynamically determine whether to prioritize retrieving textual or visual information based on the characteristics of the current subquery and contextual information. LLM's understanding of its own knowledge boundaries is also mainly limited to distinguishing between parametric knowledge and external textual knowledge. Existing mechanisms cannot effectively handle questions involving visual knowledge boundaries, such as "Do I know the content of this chart?" or "Do I need to view the details of this image?"

[0006] 3. Insufficient Complex Reasoning and Dynamic Programming Capabilities: Decomposing a complex user query involving multimodal information into a series of operable subquery sequences that include both text retrieval and visual retrieval poses a significant challenge to existing systems. Current technologies lack mature solutions for dynamically planning an optimal "retrieval chain" that includes both text retrieval and visual retrieval steps, and for adjusting subsequent strategies based on intermediate results. Summary of the Invention

[0007] To address the technical problems of weak multimodal information fusion capabilities and lack of dynamic decision-making and adaptive selection in the reasoning process in existing technologies, this invention provides a retrieval enhancement generation method and system that combines multimodal fusion and reinforcement learning. This method achieves effective fusion of multimodal information, supports intelligent selection of knowledge sources in each iteration, and optimizes the decision-making strategy and knowledge boundary cognition of large language models through imitation learning and calibration mechanisms.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] This invention discloses a retrieval enhancement generation method that combines multimodal fusion and reinforcement learning, comprising:

[0010] Receive the user's original query and combine it with the multimodal context of the current iteration step to generate a subquery based on the large language model;

[0011] The current state is formed by combining the subquery and the multimodal context. The retrieval enhancement generation task is modeled as a Markov decision process. A large language model is used to adaptively select the optimal action from a predefined action set in the current state according to the decision strategy. The action set includes existing knowledge answering for the subquery, text retrieval, visual document retrieval, and termination action.

[0012] According to the optimal action, perform the corresponding multimodal retrieval operation, fuse the acquired multimodal information, generate intermediate or final answers for subqueries, and update the multimodal context using each generated intermediate answer;

[0013] In this process, the large language model is trained and optimized offline through imitation learning and calibration chains, thereby utilizing the fine-tuned large language model for online reasoning decision-making strategies and subqueries.

[0014] As a further improvement to the above scheme, the multimodal context includes historical interaction records prior to the current iteration step and a summary of currently accumulated multimodal information; the historical interaction records include all subqueries of previous iteration steps, actions performed for the sub-steps, and corresponding intermediate answers generated;

[0015] In each iteration step, the original query and the multimodal context are formatted into a text prompt using a preset template. This text prompt is then used as input, encoded by the word segmenter of the large language model, and input into the large language model. The large language model generates a text sequence, i.e., the subquery, through autoregressive decoding.

[0016] As a further improvement to the above scheme, the process for selecting the optimal action specifically includes:

[0017] Based on the subquery and multimodal context of the current iteration step, a specific cue for action decision is constructed; the specific cue is input into the large language model, which generates a text identifier representing the selected action through autoregressive decoding; then the text identifier is parsed and mapped to a predefined action object;

[0018] In the mapping process, the text identifier is first matched precisely with the predefined standard action identifier. If the precise match fails, the text identifier is matched fuzzily with the standard action identifier by keyword extraction or string similarity algorithm. If the confidence of all matching methods is lower than a preset threshold, the preset default action is executed. If no valid action can be parsed, a specific prompt with stronger constraints or clarification requirements is sent to the large language model again.

[0019] As a further improvement to the above scheme, when performing the corresponding multimodal retrieval operation according to the optimal action, if text retrieval is performed, the subquery of the current iteration step is sent to the text retrieval machine. The text retrieval machine adopts a hybrid retrieval strategy to retrieve text fragments related to the subquery from a pre-built text knowledge base. The specific process is as follows:

[0020] The BM25 algorithm is used to perform sparse retrieval on the text knowledge base, obtaining the first few preliminarily relevant text fragments to generate sparse retrieval results. At the same time, the subquery and document fragments in the text knowledge base are encoded into high-dimensional vectors, and then a vector index is constructed to perform an approximate nearest neighbor search to obtain several text fragments whose vectors are most similar to the subquery vectors, thereby generating dense retrieval results. The sparse retrieval results and dense retrieval results are fused using the RRF reordering algorithm to obtain the set of relevant text fragments, which is the retrieved text information.

[0021] As a further improvement to the above scheme, when performing the corresponding multimodal retrieval operation according to the optimal action, if visual document retrieval is performed, the subquery of the current iteration step and the context images retrieved in the previous iteration step are sent to the visual document retrieval device. The visual document retrieval device is then invoked to retrieve page images related to the subquery from the pre-built image knowledge base. The specific process is as follows:

[0022] Using a visual encoder, each page image in the image knowledge base is pre-encoded into a format that is... M A sequence of image patch embedding vectors When performing an online search, the subquery will be encoded as... N A sequence of query token embedding vectors Then, a post-interaction matching mechanism is used to calculate the relevance score between the subquery and each candidate page image. A maximum similarity aggregation strategy is adopted to obtain the set of the top few most relevant page images, i.e., the retrieved visual information. The formula for calculating the relevance score is as follows:

[0023] ;

[0024] Subquery With page images The relevance score; For the first j Image blocks The encoded embedding vector, ; For the first i An embedded vector formed by encoding a query token. ; This indicates the calculation of cosine similarity.

[0025] As a further improvement to the above scheme, the process of generating the intermediate or final answer of the subquery includes:

[0026] The textual and / or visual information obtained in the current iteration step is fused with the information in the multimodal context to form a complete context for answer generation in the current iteration step;

[0027] If the optimal action selected in the current iteration step is not the terminating action, then the subquery and the complete context are input into the large language model. The large language model generates an answer or relevant information summary for the subquery through autoregressive decoding, which serves as an intermediate answer.

[0028] If the optimal action selected in the current iteration step is the termination action, or the preset maximum number of iterations is reached, then the original query and the complete context are input into the large language model. The large language model generates the answer to the original query, i.e., the final answer, through autoregressive decoding.

[0029] Before inputting the complete context into the large language model, the semantic relevance of each information fragment in the complete context to the current query to be answered is evaluated using the cosine similarity of the embedding vectors. Then, information fragments are selected from high to low according to the semantic relevance scores, and the latest interaction history is retained first, until the maximum input length acceptable to the large language model is reached. Any part exceeding the limit is truncated, thereby constructing a prompt containing the complete context and the current query to be answered, which is used as the input to the large language model.

[0030] As a further improvement to the above scheme, the specific process of offline training and optimization of the large language model includes:

[0031] Construct a training dataset, in which each sample includes the original query and its corresponding standard reference answer;

[0032] The ternary tree search algorithm is used for each sample in the training dataset to automatically explore multiple decision paths from the initial state based on the original query to the final answer that can generate a high degree of matching with the standard reference answer; where each node of the tree represents a state and each edge represents an action;

[0033] The total reward for each decision path is evaluated using a predefined reward function;

[0034] Select the set of decision paths with the highest total reward, and extract the state-optimal action pairs from them. State-Optimal Subquery Pair This constitutes an imitation learning dataset. ;in, It indicates the state of the current iteration step, including the original query, historical interaction records, and currently accumulated multimodal context information; Indicates the state The optimal action selected based on the decision-making strategy; Indicates the execution of an action The next state obtained afterward contains the updated multimodal context information and the intermediate answer; Indicates the state The optimal subquery generated below serves as the semantic expression of the current subtask, guiding subsequent retrieval or generation operations.

[0035] Using imitation learning datasets Supervised fine-tuning is performed on the modules of the large language model responsible for decision-making strategies and subquery generation; specifically, for the decision-making strategy generation task, the state is... The representation is used as input to determine the optimal action. As labels, the cross-entropy loss function is used for training; for the subquery generation task, the state is used... The representation is used as input to select the optimal subquery. The target sequence is used for training with the loss function of a standard sequence-to-sequence model.

[0036] As a further improvement to the above scheme, the expression of the predefined reward function is:

[0037] ;

[0038] In the formula, Decision path Total reward; The weighting coefficient for the accuracy of the answer; Decision path The final answer produced; This is the standard reference answer; For precise matching functions, if and If they are exactly the same, the value is 1; otherwise, the value is 0. Decision path The first in One action; For action Cost weighting; For action The cost value.

[0039] As a further improvement to the above scheme, the specific process of offline training and optimization of the large language model also includes:

[0040] Several subqueries and state pairs were obtained by sampling from the validation set, and the large language model after the imitation learning phase was used. Generate preference data; where, for each subquery and state pair , respectively using Subsequent interaction sequences and intermediate answers are generated through three decision paths. The relative merits of the three decision paths are determined by evaluating the contribution of the intermediate answers generated by each decision path to the standard reference answer. The three decision paths are answering based on existing knowledge, performing text retrieval, and performing visual document retrieval. The evaluation criteria for contribution are as follows: calculate three indicators for the intermediate answers generated by each decision path and the standard reference answer, namely BLEU score, ROUGE-L, and EM score. After combining the three indicators using a weighted average method, the merits of the decision paths are ranked according to the weighted scores.

[0041] For each subquery and state pair Construct a preference pair The preference dataset is composed of all preference pairs. ;in, for The textual representation of The subsequent sequence of interactions and answers generated for the decision path that is evaluated as the better one; The sequence of subsequent interactions and answers generated for a decision path that was evaluated as suboptimal;

[0042] Using preference datasets For large language models Direct preference optimization training is performed to make its decision strategy tend to choose multimodal decision paths that bring better results; the direct preference optimization loss function is as follows:

[0043] ;

[0044] In the formula, To optimize loss for direct preference, The current strategy to be optimized; For reference strategy; For preference datasets Expectation of the middle sample; For input context; The preferred response sequence; The sequence of responses that are not preferred; For the Sigmoid function; This is a temperature parameter used to control the degree of deviation from the reference strategy.

[0045] This invention also discloses a retrieval enhancement generation system based on multimodal fusion and reinforcement learning collaboration, applying the retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration as described above; the retrieval enhancement generation system includes:

[0046] The query understanding and iterative decomposition module is used to receive the original query input by the user and generate subqueries based on the large language model by combining the multimodal context of the current iteration step.

[0047] An adaptive retrieval decision module is used to combine the subquery and the multimodal context to form the current state, model the retrieval enhancement generation task as a Markov decision process, and use a large language model to adaptively select the optimal action from a predefined action set in the current state according to the decision strategy; wherein, the action set includes existing knowledge answering for the subquery, text retrieval, visual document retrieval, and termination action;

[0048] The information fusion and generation module is used to perform corresponding multimodal retrieval operations according to the optimal action, fuse the acquired multimodal information, generate intermediate or final answers for subqueries, and update the multimodal context using each generated intermediate answer;

[0049] The decision optimization module is used to train and optimize the large language model offline through imitation learning and calibration chains, thereby using the fine-tuned large language model to reason about decision strategies and subqueries online.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] 1. Enhanced multimodal information processing and fusion capabilities: By integrating visual retrieval capabilities into the iterative framework, the system can effectively acquire and fuse information from both textual and visual modalities, significantly improving its understanding and question-answering capabilities for complex documents containing charts, images, and other elements.

[0052] A smarter and more adaptive iterative reasoning strategy: Based on the Markov decision process-based decision framework and the extended multimodal imitation learning and calibration chain, the large language model can learn to dynamically select the optimal knowledge source (parametric knowledge, text retrieval, or visual retrieval) according to the current situation at each reasoning step, thereby achieving more efficient and accurate complex query processing.

[0053] Enhancing the ability to solve complex problems and the quality of answers: By progressively decomposing problems, retrieving multimodal information, and employing iterative reasoning, it can better solve complex problems requiring multi-hop reasoning, information synthesis, and cross-modal understanding, generating more comprehensive, accurate, and reliable answers. The iterative subquery decomposition and fusion feedback mechanism supports the dynamic generation of sub-problems, feeding back preceding information to the next round of retrieval and generation, thus realizing a human-like reasoning chain.

[0054] 2. Optimized Knowledge Boundary Recognition: The multimodal calibration chain of this invention enables large models not only to determine "whether they know it" but also "whether searching for text or images is more effective for things they don't know," thereby reducing unnecessary searches and erroneous information dependencies. It innovatively introduces "modal preference comparison data" to optimize the LLM's recognition of decision boundaries under different modalities.

[0055] 3. Automated generation of high-quality training data: The extended ternary tree search mechanism of this invention can automatically generate high-quality training trajectories containing multimodal decision paths, reducing the reliance on complex reasoning processes with manual annotation and constructing high-quality imitation learning data. Attached Figure Description

[0056] Figure 1 This is a flowchart of the retrieval enhancement generation method that combines multimodal fusion and reinforcement learning in Embodiment 1 of the present invention.

[0057] Figure 2 This is an architecture diagram of the retrieval enhancement generation system that combines multimodal fusion and reinforcement learning in Embodiment 1 of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] Please see Figure 1 and Figure 2 This embodiment provides a retrieval enhancement generation method that combines multimodal fusion and reinforcement learning. The corresponding system mainly includes four core modules: a query understanding and iterative decomposition module, an adaptive retrieval decision module, an information fusion and generation module, and a multimodal decision optimization module based on imitation learning and calibration. The following section will explain these modules and their corresponding methods in detail.

[0061] (1) Query understanding and iterative decomposition module

[0062] This module is used to receive raw query input from the user. In conjunction with the multimodal context of the current iteration step, subqueries are generated based on the large language model.

[0063] The multimodal context Includes historical interaction records prior to the current iteration step. and currently accumulated multimodal information summaries The historical interaction record includes all subqueries of previous iteration steps, actions performed for the substeps, and corresponding intermediate answers generated.

[0064] In each iteration step, the original query is... With multimodal context Format into a complete text prompt using a preset template. The prompt template is designed as follows: "The original query was: [Enter here]" Based on the following reasoning steps already taken and the known information: [Fill in the formatted text here] and To ultimately answer the original query, what is the simplest, clearest, and most actionable sub-question to ask next? Please provide the sub-question directly:

[0065] In this embodiment, the core of the iterative decomposition query is a large language model fine-tuned based on Llama-3.1-8B-Instruct. Subquery generation breaks down complex problems into a series of simpler questions that are easier to answer through retrieval or parametric knowledge. The query understanding and iterative decomposition module receives the original query input by the user. In each iteration step t Based on the current multimodal context and the original query This generates the next atomic, operable subquery. This process is iterative. The generation will take into account The corresponding processing result is expressed as follows:

[0066] ;

[0067] In the formula, This indicates the decomposition function of LLM. For LLM parameters.

[0068] The Llama-3.1-8B-Instruct model described above can be loaded using HuggingFace's Transformers library. This includes loading its corresponding tokenizer and model weights, which are obtained through fine-tuning training via the decision optimization module. This will enable the text prompts... As input, the data is encoded by the word segmenter of the large language model and then fed into the large language model, which generates a text sequence, i.e., a subquery, through autoregressive decoding. During the generation process, the temperature was set to 0.7. The sampling parameter is set to 0.9 to control the creativity and focus of the generated subqueries. After generation, the output is post-processed to extract the clean subquery text.

[0069] This method enables the system to break down complex problems step by step, reducing information overload in a single process and improving the systematicness and controllability of problem solving.

[0070] Furthermore, this method also possesses context-dependent characteristics: the generation of subqueries is deeply dependent on previous reasoning steps and acquired multimodal information, ensuring the coherence and goal-orientation of the reasoning chain. This avoids generating subqueries that are irrelevant to or repetitive with the current reasoning goal, improving the efficiency and relevance of the entire reasoning process.

[0071] (2) Adaptive retrieval decision module

[0072] This module is used to combine the subquery and the multimodal context to form the current state, model the retrieval enhancement generation task as a Markov decision process, and use a large language model to adaptively select the optimal action from a predefined action set in the current state according to the decision strategy; the action set includes existing knowledge answering for the subquery, text retrieval, visual document retrieval, and termination action.

[0073] In this embodiment, the process of selecting the optimal action specifically includes:

[0074] Subquery based on the current iteration step With multimodal context Build a specific cue for action decision This suggestion clearly lists the key information of the current subquery and context, and provides predefined, clear multimodal action options for LLM to choose from. The suggestion template is designed as: "The current sub-problem to be solved is: [Enter here]" Based on the context information currently available: [Fill in here] Please select the most appropriate action from the following to answer the current subquestion (please strictly return the English identifier of the selected action): 1. (Answer directly using existing knowledge); 2. (Retrieve relevant text); 3. (Retrieve relevant images or visual documents); 4. (Sufficient information is available; end the search and generate the final answer). Your decision is:

[0075] Specific prompts are input into a large language model, which then generates text identifiers representing the selected action through autoregressive decoding, such as " The text identifier is then parsed and mapped to a predefined action object. For example, if the LLM outputs " ", then is set to perform the action of visual retrieval and carry the parameter .

[0076] Among them, in the mapping process, first perform an exact string match between the text identifier and the predefined standard action identifier. If the exact match fails, then through keyword extraction (for example, if "text" or "文本" is recognized, it tends to ; if "visual", "image", "图像", or "图表" is recognized, it tends to ) or a string similarity algorithm (such as Levenshtein distance) to perform a fuzzy match between the text identifier and the standard action identifier; if the confidence of all matching methods is lower than the preset threshold (set to 0.7), then perform the preset default action (conservatively select ), if no valid action can be parsed, then send a specific prompt with stronger constraints or clarification requirements to the large language model again. The finally determined action will be used to guide the subsequent information retrieval or generation steps. If the action requires parameters (such as a sub-query for retrieval), then will be passed as a parameter.

[0077] Multi-modal state representation and action space: The adaptive retrieval decision module receives from the query understanding and iterative decomposition module and obtains from the information fusion and generation module to form the state ; the adaptive retrieval decision module selects an action according to the current state and its internally learned decision-making strategy, and according to the decision-making action , passes to the corresponding text retriever, visual document retriever, or information fusion and generation module. Its decision-making logic is learned by the decision optimization module, aiming to maximize the expected cumulative reward.

[0078] The adaptive retrieval decision module formalizes the RAG process as a Markov decision process MDP, and the expression is as follows:

[0079] ;

[0080] In the formula, S represents the state space, representing the complete information environment of the system at each decision-making moment; is the extended multi-modal action space. At each state , the LLM (as the policy The executor selects an action from a predefined set of actions. The predefined set of actions contains knowledge-based answers that depend on LLM parameters. of ,right Perform text retrieval ,right Perform visual document retrieval The flag indicates that the iteration has terminated and the system is ready to generate the final answer. The multimodal action space empowers the system to flexibly select the most suitable knowledge source based on current information needs, breaking through the limitations of the single retrieval mode of traditional RAG systems; P This represents the state probability transition function, which describes the state transition. S t Next action a t Afterwards, the system transitions to a new state. S t+1 The probability distribution; R This represents the reward function, used to evaluate the immediate benefit of performing an action in a specific state; γ The discount factor is a constant between 0 and 1 used to adjust the weight of future rewards in the current decision. γ A value close to 1 indicates a focus on long-term gains and a willingness to accept immediate costs for future returns. γ A value close to 0 indicates a greater focus on immediate rewards and a tendency towards short-term, efficient behavior.

[0081] Strategic decision-making: Action selection is based on optimizing the policy learned by the LLM in the decision optimization module.

[0082]

[0083] in In the state The action to choose. For parameters The policy network represents the state. The probability distribution of each action is given below. This strategy aims to maximize the expected cumulative reward, i.e., the state-value function. :

[0084] ;

[0085] in This represents the expectation under strategy π; As a discount factor, ; The instant reward to be obtained at a future time step t+k+1.

[0086] The optimization goal is to find the optimal strategy. , making This holds true for all states s and all policies π. Policies trained through reinforcement learning make the decision-making process more intelligent and adaptive, dynamically adjusting information acquisition methods according to specific problem scenarios, thereby improving the accuracy and necessity of retrieval and avoiding unnecessary retrieval overhead.

[0087] As an intelligent agent, LLM can adaptively select from a multimodal action space that includes parameter knowledge utilization, text information retrieval, and visual document information retrieval based on page images (using the principle of post-interaction multi-vector matching) according to the current state at each decision step, and iteratively perform query understanding and decomposition.

[0088] (3) Information fusion and generation module

[0089] This module is used to perform corresponding multimodal retrieval operations based on the optimal action, fuse the acquired multimodal information, generate intermediate or final answers for subqueries, and update the multimodal context using each generated intermediate answer.

[0090] Text retrieval and visual document retrieval can be performed in parallel. If a text retrieval is performed... Then the subquery of the current iteration step will be... The text is sent to a text retrieval tool that employs a hybrid retrieval strategy, drawing from a pre-built text knowledge base. Search and subqueries The relevant text fragments, the specific process is as follows:

[0091] The BM25 algorithm (implemented via the Pyserini library) of Jingdian is used to perform sparse retrieval on a text knowledge base (a collection of paragraphs from Wikipedia, with an inverted index already built), retrieving the top-N1 most relevant text fragments to generate sparse search results; simultaneously, the intfloat / e5-large-v2 model loaded from HuggingFace is used to process subqueries. Document fragments from the text knowledge base are encoded into high-dimensional vectors. Then, a vector index is constructed using the FAISS library to perform an approximate nearest neighbor search, obtaining several (Top-N2) text fragments whose vectors are most similar to the subquery vector, thus generating dense search results. Finally, the sparse and dense search results can be fused using the RRF (Reciprocal Rank Fusion) algorithm to obtain the final Top-K. text The retrieved text information is a collection of relevant text fragments.

[0092] If performing visual document retrieval Then the subquery of the current iteration step will be... and the context image retrieved in previous iterations. Send to the visual document retrieval unit, which then invokes the visual document retrieval unit to retrieve information from a pre-built image knowledge base. Search and subqueries The relevant page images, the specific process is as follows:

[0093] Access the preprocessed image library, the images on each page of the library. It has been segmented using the google / siglip-so400m-patch14-384 visual model and its corresponding image processor, AutoImageProcessor. M Each image patch (paches) Encoded as an embedding vector These embedding vectors are stored using FAISS and form a sequence. When searching online, you can use the intfloat / e5-large-v2 model in text retrieval to subquery. Encoded as by N A sequence of query token embedding vectors Then, a post-interaction matching mechanism is used to calculate the relevance score between the subquery and each candidate page image. A maximum similarity aggregation (MaxSim) strategy is employed to obtain the top-3 most relevant page image sets, i.e., the retrieved visual information. The formula for calculating the relevance score is as follows:

[0094] ;

[0095] Subquery With page images The relevance score; For the first j Image blocks The encoded embedding vector, ; For the first i An embedded vector formed by encoding a query token. ; This indicates the calculation of cosine similarity.

[0096] Multimodal retrieval enables the system to directly utilize visual elements in documents (such as charts and illustrations) to obtain deeper information that cannot be obtained through pure text retrieval, significantly enhancing the ability to understand multimodal documents.

[0097] The process of generating the intermediate or final answer to the subquery includes:

[0098] The text information obtained in the current iteration step and / or visual information (For each page image in the visual information, a concise text description can be generated for the image using the LLaVA-1.5 model. The text description output by LLaVA will be normalized to unify the embedding space.) This is combined with the previously accumulated multimodal context. The information is input into a fusion layer with a cross-modal attention mechanism (this fusion layer adopts a Transformer structure and introduces cross-modality attention during the encoding stage, combining visual description embeddings and textual information as input), forming a complete context for answer generation in the current iteration step. ,Right now:

[0099] ;

[0100] In the formula, This is a multimodal information fusion function; To analyze the original visual retrieval results The processed representation (such as text descriptions or compatible embeddings) is then used. Information fusion ensures that information from different sources and modalities can be uniformly understood and utilized by LLM, laying the foundation for generating high-quality, information-rich answers and improving the problem of information fragmentation.

[0101] Answer generation: based on subqueries and full context Using LLM to generate intermediate answers Or the final answer In this embodiment, the answer generation can employ the GPT-4o model called via the OpenAI API. Before inputting the complete context into the GPT-4o model, the semantic relevance of each information fragment in the complete context to the current query to be answered is evaluated using the cosine similarity of the embedding vectors. Then, information fragments are selected from high to low according to their semantic relevance scores, with priority given to retaining the latest interaction history, until the maximum input length acceptable to the large language model is reached. Any excess is truncated, thereby constructing a hint containing the complete context and the current query to be answered. This is used as the input to the GPT-4o model. The template is designed to "Refer to the following information: [Enter formatted text here]" Please answer the question: [Fill in here] or The answer is: "". Upon receiving this prompt, the GPT-4o model generates the corresponding intermediate answer through autoregressive decoding. Or the final answer .

[0102] In each iteration step t In the process, if the adaptive retrieval decision module selects the optimal action... Not the termination action (i.e., the iteration is not yet complete), then the subquery and full context Input a large language model, which generates paired subqueries through autoregressive decoding. The answer or related information summary, as an intermediate answer This process can be represented as:

[0103] ;

[0104] In the formula, This indicates that LLM performs the function of generating intermediate answers; The context will be used to update the next iteration step. .

[0105] In each iteration step t In the process, if the optimal action selected by the adaptive retrieval decision module is the termination action... If the preset maximum number of iterations is reached, the final answer will be generated. At this point, the adaptive retrieval decision module will process the original query... and the final complete context accumulated until the iteration terminates. The input is a large language model, which generates the answer to the original query, i.e., the final answer, through autoregressive decoding. This process can be represented as:

[0106] ;

[0107] In the formula, This represents the function of LLM to perform the final answer generation.

[0108] By fully utilizing the fused multimodal context, the generated answers are more factual, relevant, and in-depth, effectively reducing the "illusion" phenomenon of the model.

[0109] (4) Decision optimization module

[0110] This module is used to train and optimize large language models offline through imitation learning and calibration chains, thereby enabling online inference decision strategies and subqueries using the fine-tuned large language models.

[0111] The core task of the decision optimization module is to train and optimize the decision-making strategy of LLM. This enables it to perform efficient and accurate multimodal iterative inference. The specific process of offline training and optimization of the large language model includes:

[0112] Construct a training dataset where each sample includes the original query and its corresponding standard reference answer.

[0113] Automated multimodal trajectory data synthesis: employing a ternary tree search algorithm (branch correspondence) The core actions (such as parametric knowledge answering, text retrieval, and visual document retrieval) are performed on each sample in the training dataset. Automatically explores from an initial state based on the original query to being able to generate a standard reference answer. Multiple trajectories of the highly matched final answer That is, the decision path; where each node of the tree represents a state and each edge represents an action.

[0114] The total reward for each decision path is evaluated using a predefined reward function; the expression for the predefined reward function is:

[0115] ;

[0116] In the formula, Decision path Total reward; This is a weighting factor for the accuracy of the answer, typically set to 0.7; Decision path The final answer produced; This is the standard reference answer (the gold standard answer); For precise matching functions, if and If they are exactly the same, the value is 1; otherwise, the value is 0. Decision path The first in One action; For action The cost weight is set to 0.3; For action The cost value; It is set to 0.002 because it mainly consumes the LLM's own inference capabilities; Setting it to 0.1 reflects the overhead of text encoding, index lookup, and result processing; It is set to 0.15, which is the highest setting, because it involves image encoding and complex visual feature matching; Set to 0.

[0117] Select the set of decision paths with the highest total reward, and extract the state-optimal action pairs from them. State-Optimal Subquery Pair This constitutes an imitation learning dataset. ;in, It indicates the state of the current iteration step, including the original query, historical interaction records, and currently accumulated multimodal context information; Indicates the state The optimal action selected based on the decision-making strategy; Indicates the execution of an action The next state obtained afterward contains the updated multimodal context information and the intermediate answer; Indicates the state The optimal subquery generated is used as the semantic representation of the current subtask to guide subsequent retrieval or generation operations. This method automates the generation of high-quality training data, reduces reliance on expensive manually labeled data, and improves the scalability of training.

[0118] Using imitation learning datasets Supervised fine-tuning is performed on the modules responsible for decision-making strategies and subquery generation in the large language model.

[0119] For the decision strategy generation task, the state is... The representation is used as input to determine the optimal action. As labels, the cross-entropy loss function is used for training, and the loss function is as follows:

[0120] ;

[0121] In the formula, The loss function is used to imitate the learning strategy part; These are state-action pairs extracted from the optimal trajectory. For the model in state Choose the optimal action The probability of.

[0122] For the subquery generation task, the status will be... The representation is used as input to select the optimal subquery. The target sequence is used for training, employing a standard sequence-to-sequence model loss function. Both tasks are optimized simultaneously using a multi-task learning framework. Training at this stage utilizes HuggingFace's Trainer API.

[0123] To further enhance the ability of LLM to distinguish the applicability boundaries of different knowledge modalities, several sub-queries and state pairs were sampled from the validation set, and a large language model after the imitation learning phase was used. Generate preference data The validation set is a reserved portion of the training dataset that is not used in the model training process during the imitation learning phase. It is used only for subsequent preference data generation and evaluation to ensure the independence and openness of the validation data. For each subquery and state pair... Forced This study explores the subsequent interaction sequences and intermediate answers generated through three decision paths. The relative merits of the three decision paths are determined by evaluating the contribution of the intermediate answers generated by each path to the standard reference answer. The three decision paths are: answering based on existing knowledge, performing text retrieval, and performing visual document retrieval. Each path uses only a single category of action when generating intermediate answers, without mixing different retrieval methods to ensure comparability and independence between paths. The evaluation criteria for contribution are: calculating three indicators for the intermediate answers generated by each decision path and the standard reference answer, namely BLEU score, ROUGE-L score, and EM score; using a weighted average method to synthesize the three indicators; and ranking the decision paths according to the weighted scores.

[0124] For each subquery and state pair Construct a preference pair The preference dataset is composed of all preference pairs. ;in, for The textual representation of The subsequent sequence of interactions and answers generated for the decision path that is evaluated as the better one; The sequence of subsequent interactions and answers generated for a decision path that was evaluated as suboptimal;

[0125] Using preference datasets For large language models Direct preference optimization (DPO) training is performed to make its decision strategy tend to choose multimodal decision paths that bring better results; the direct preference optimization loss function is as follows:

[0126] ;

[0127] In the formula, To optimize loss for direct preference, The current strategy to be optimized; For reference strategy; For preference datasets Expectation of the middle sample; For input context; The winning response sequence; The sequence of unfavorable (losing) responses; For the Sigmoid function; This is a temperature parameter used to control the degree of deviation from the reference strategy.

[0128] Direct preference optimization training can be implemented using the DPOTrainer class in the Transformer Reinforcement Learning (TRL) library. When configuring DPOTrainer, you will... As the model to be trained, the reference model yes An initial copy (its parameters are fixed before DPO training begins). All input contexts and response sequences will be uniformly formatted to fit the DPO input format. In the DPO loss function, the temperature parameter... Set to 0.1. Through DPO training, the LLM's decision logic will adjust towards a direction that favors choosing multimodal decision paths that yield better results. After training, the final optimized LLM is obtained and used for online inference. This embodiment... LlaMA2-7B can be used, which adapts to multimodal decision generation tasks through imitation learning and DPO optimization during the offline training phase.

[0129] By optimizing training through direct preference, the accuracy of LLM's perception of its knowledge boundaries in different modalities is significantly improved. This enables it to more intelligently determine when to rely on internal knowledge and when and how (text or vision) to utilize external knowledge, thereby optimizing the necessity and effectiveness of retrieval and further improving the reliability of the answers.

[0130] In summary, after receiving the user's original query, the RAG system of this invention first generates a subquery through the query understanding and iterative decomposition module. Subsequently, the adaptive retrieval decision module selects an optimal multimodal action for this subquery based on the current state. The information fusion and generation module executes this action, fusing the retrieved information to generate an intermediate answer for the subquery while updating the multimodal context. This process of "subquery generation - multimodal decision - retrieval - information fusion - intermediate answer generation" is iterative. At each step, the LLM's decision logic is shaped by an offline-trained decision optimization module. When the adaptive retrieval decision module determines that sufficient information has been collected or a preset termination condition has been met, the information fusion and generation module generates the final answer for the original query based on the final complete context. Through this iterative, multimodal-aware processing flow, this invention can more deeply understand user intent, more comprehensively acquire the required information, and thus generate a higher-quality response.

[0131] To verify the effectiveness of the method of the present invention, the following experiments are provided in this embodiment:

[0132] I. Experimental Setup

[0133] Experimental task: Multi-round question-answering retrieval enhancement generation task, covering complex queries in mixed text and image scenarios, such as document understanding, chart question answering, and visual reasoning.

[0134] Dataset source: A test set containing 1000 multimodal raw queries (text and related images) and manually labeled answers was constructed; all data included explicit reference answers for accuracy evaluation.

[0135] Evaluation metrics: Accuracy metrics for generated answers: BLEU, ROUGE-L, EM (Exact Match); Multimodal decision rationality metrics: Matching degree between action selection and retrieval path.

[0136] Comparison method:

[0137] Baseline 1: Question-and-answer based solely on LLM, without introducing any retrieval module;

[0138] Baseline 2: Traditional RAG (Text Retrieval Enhancement);

[0139] Baseline 3: Multimodal RAG (Image and Text Retrieval Enhancement), but without introducing reinforcement learning and path selection;

[0140] The method of this invention combines reinforcement learning modeling, subquery generation, adaptive action selection, imitation learning, and DPO optimization.

[0141] II. Experimental Results

[0142] As shown in Table 1, the action rationality rate represents the proportion of actions selected by the model that are consistent with the actual optimal path, reflecting the model's path decision-making ability.

[0143] Table 1: Comparison of Evaluation Indicators for Each Method

[0144] ;

[0145] Note: In Table 1, the upward arrow "↑" indicates that the higher the result of the indicator, the better, and the downward arrow "↓" indicates that the lower the result of the indicator, the better.

[0146] III. Experimental Conclusions

[0147] Experimental results show that, compared with existing methods, the method of this invention achieves significant improvements in accuracy metrics (BLEU, ROUGE-L, EM), especially demonstrating higher intelligence and robustness in multimodal decision path selection and execution efficiency. In particular, by introducing a reinforcement learning framework and DPO-optimized training, the large language model can more effectively select the optimal action path under different states, thereby improving the overall performance of the retrieval augmentation generation system. Therefore, the method of this invention can significantly improve the retrieval generation quality of large language models in complex, multimodal scenarios, and has significant technical effects and promotional value.

[0148] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A retrieval enhancement generation method based on multimodal fusion and reinforcement learning, characterized in that, include: Receive the user's original query and combine it with the multimodal context of the current iteration step to generate a subquery based on the large language model; The current state is formed by combining the subquery and the multimodal context. The retrieval enhancement generation task is modeled as a Markov decision process. A large language model is used to adaptively select the optimal action from a predefined action set in the current state according to the decision strategy. The action set includes existing knowledge answering for the subquery, text retrieval, visual document retrieval, and termination action. According to the optimal action, perform the corresponding multimodal retrieval operation, fuse the acquired multimodal information, generate intermediate or final answers for subqueries, and update the multimodal context using each generated intermediate answer; In this process, the large language model is trained and optimized offline through imitation learning and calibration chains, thereby utilizing the fine-tuned large language model for online reasoning decision-making strategies and subqueries.

2. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 1, characterized in that, The multimodal context includes historical interaction records prior to the current iteration step and a summary of currently accumulated multimodal information; the historical interaction records include all subqueries of previous iteration steps, actions performed for the sub-steps, and corresponding intermediate answers generated; In each iteration step, the original query and the multimodal context are formatted into a text prompt using a preset template. This text prompt is then used as input, encoded by the word segmenter of the large language model, and input into the large language model. The large language model generates a text sequence, i.e., the subquery, through autoregressive decoding.

3. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 1, characterized in that, The process of selecting the optimal action specifically includes: Based on the subquery and multimodal context of the current iteration step, a specific cue for action decision is constructed; the specific cue is input into the large language model, which generates a text identifier representing the selected action through autoregressive decoding; then the text identifier is parsed and mapped to a predefined action object; In the mapping process, the text identifier is first matched precisely with the predefined standard action identifier. If the precise match fails, the text identifier is matched fuzzily with the standard action identifier by keyword extraction or string similarity algorithm. If the confidence of all matching methods is lower than a preset threshold, the preset default action is executed. If no valid action can be parsed, a specific prompt with stronger constraints or clarification requirements is sent to the large language model again.

4. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 1, characterized in that, When performing the corresponding multimodal retrieval operation based on the optimal action, if text retrieval is performed, the subquery of the current iteration step is sent to the text retrieval machine. The text retrieval machine adopts a hybrid retrieval strategy to retrieve text fragments related to the subquery from a pre-built text knowledge base. The specific process is as follows: The BM25 algorithm is used to perform sparse retrieval on the text knowledge base, obtaining the first few preliminarily relevant text fragments to generate sparse retrieval results. At the same time, the subquery and document fragments in the text knowledge base are encoded into high-dimensional vectors, and then a vector index is constructed to perform an approximate nearest neighbor search to obtain several text fragments whose vectors are most similar to the subquery vectors, thereby generating dense retrieval results. The sparse retrieval results and dense retrieval results are fused using the RRF reordering algorithm to obtain the set of relevant text fragments, which is the retrieved text information.

5. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 4, characterized in that, When performing the corresponding multimodal retrieval operation based on the optimal action, if visual document retrieval is performed, the subquery of the current iteration step and the context images retrieved in the previous iteration step are sent to the visual document retrieval device. The visual document retrieval device is then invoked to retrieve page images related to the subquery from the pre-built image knowledge base. The specific process is as follows: Using a visual encoder, each page image in the image knowledge base is pre-encoded into a format that is... M A sequence of image patch embedding vectors When performing an online search, the subquery will be encoded as... N A sequence of query token embedding vectors ; Then, a post-interaction matching mechanism is used to calculate the relevance score between the subquery and each candidate page image. A maximum similarity aggregation strategy is adopted to obtain the set of the top few most relevant page images, i.e., the retrieved visual information. The formula for calculating the relevance score is as follows: ； Subquery With page images The relevance score; For the first j Image blocks The encoded embedding vector, ; For the first i An embedded vector formed by encoding a query token. ; This indicates the calculation of cosine similarity.

6. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 5, characterized in that, The process of generating the intermediate or final answer to the subquery includes: The textual and / or visual information obtained in the current iteration step is fused with the information in the multimodal context to form a complete context for answer generation in the current iteration step; If the optimal action selected in the current iteration step is not the terminating action, then the subquery and the complete context are input into the large language model. The large language model generates an answer or relevant information summary for the subquery through autoregressive decoding, which serves as an intermediate answer. If the optimal action selected in the current iteration step is the termination action, or the preset maximum number of iterations is reached, then the original query and the complete context are input into the large language model. The large language model generates the answer to the original query, i.e., the final answer, through autoregressive decoding. Before inputting the complete context into the large language model, the semantic relevance of each information fragment in the complete context to the current query to be answered is evaluated using the cosine similarity of the embedding vectors. Then, information fragments are selected from high to low according to the semantic relevance scores, and the latest interaction history is retained first, until the maximum input length acceptable to the large language model is reached. Any part exceeding the limit is truncated, thereby constructing a prompt containing the complete context and the current query to be answered, which is used as the input to the large language model.

7. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 1, characterized in that, The specific process of offline training and optimization of the large language model includes: Construct a training dataset, in which each sample includes the original query and its corresponding standard reference answer; The ternary tree search algorithm is used for each sample in the training dataset to automatically explore multiple decision paths from the initial state based on the original query to the final answer that can generate a high degree of matching with the standard reference answer; where each node of the tree represents a state and each edge represents an action; The total reward for each decision path is evaluated using a predefined reward function; Select the set of decision paths with the highest total reward, and extract the state-optimal action pairs from them. State-Optimal Subquery Pair This constitutes an imitation learning dataset. ;in, It indicates the state of the current iteration step, including the original query, historical interaction records, and currently accumulated multimodal context information; Indicates the state The optimal action selected based on the decision-making strategy; Indicates the execution of an action The next state obtained afterward contains the updated multimodal context information and the intermediate answer; Indicates the state The optimal subquery generated below serves as the semantic expression of the current subtask, guiding subsequent retrieval or generation operations. Using imitation learning datasets Supervised fine-tuning is performed on the modules of the large language model responsible for decision-making strategies and subquery generation; specifically, for the decision-making strategy generation task, the state is... The representation is used as input to determine the optimal action. As labels, the cross-entropy loss function is used for training; for the subquery generation task, the state is used... The representation is used as input to select the optimal subquery. The target sequence is used for training with the loss function of a standard sequence-to-sequence model.

8. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 7, characterized in that, The expression for the predefined reward function is: ； In the formula, Decision path Total reward; The weighting coefficient for the accuracy of the answer; Decision path The final answer produced; This is the standard reference answer; For precise matching functions, if and If they are exactly the same, the value is 1; otherwise, the value is 0. Decision path The first in One action; For action Cost weighting; For action The cost value.

9. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 7, characterized in that, The specific process of offline training and optimization of the large language model also includes: Several subqueries and state pairs were obtained by sampling from the validation set, and the large language model after the imitation learning phase was used. Generate preference data; where, for each subquery and state pair , respectively using Subsequent interaction sequences and intermediate answers are generated through three decision paths. The relative merits of the three decision paths are determined by evaluating the contribution of the intermediate answers generated by each decision path to the standard reference answer. The three decision paths are answering based on existing knowledge, performing text retrieval, and performing visual document retrieval. The evaluation criteria for contribution are as follows: calculate three indicators for the intermediate answers generated by each decision path and the standard reference answer, namely BLEU score, ROUGE-L, and EM score. After combining the three indicators using a weighted average method, the merits of the decision paths are ranked according to the weighted scores. For each subquery and state pair Construct a preference pair The preference dataset is composed of all preference pairs. ;in, for The textual representation of The subsequent sequence of interactions and answers generated for the decision path that is evaluated as the better one; The sequence of subsequent interactions and answers generated for a decision path that was evaluated as suboptimal; Using preference datasets For large language models Direct preference optimization training is performed to make its decision strategy tend to choose multimodal decision paths that bring better results; the direct preference optimization loss function is as follows: ； In the formula, To optimize loss for direct preference, The current strategy to be optimized; For reference strategy; For preference datasets Expectation of the middle sample; For input context; The preferred response sequence; The sequence of responses that are not preferred; For the Sigmoid function; This is a temperature parameter used to control the degree of deviation from the reference strategy.

10. A retrieval enhancement generation system based on multimodal fusion and reinforcement learning, characterized in that, The method for retrieval enhancement generation that combines multimodal fusion and reinforcement learning as described in any one of claims 1 to 9 is applied. The retrieval enhancement generation system includes: The query understanding and iterative decomposition module is used to receive the original query input by the user and generate subqueries based on the large language model by combining the multimodal context of the current iteration step. An adaptive retrieval decision module is used to combine the subquery and the multimodal context to form the current state, model the retrieval enhancement generation task as a Markov decision process, and use a large language model to adaptively select the optimal action from a predefined action set in the current state according to the decision strategy; wherein, the action set includes existing knowledge answering for the subquery, text retrieval, visual document retrieval, and termination action; The information fusion and generation module is used to perform corresponding multimodal retrieval operations according to the optimal action, fuse the acquired multimodal information, generate intermediate or final answers for subqueries, and update the multimodal context using each generated intermediate answer; The decision optimization module is used to train and optimize the large language model offline through imitation learning and calibration chains, thereby using the fine-tuned large language model to reason about decision strategies and subqueries online.

Images