Multi-modal fusion and reinforcement learning collaborative retrieval enhancement generation method and system

Abstract

The invention relates to the technical field of information retrieval, and discloses a multi-modal fusion and reinforcement learning collaborative retrieval enhancement generation method and system. The method comprises the following steps: receiving an original query input by a user, and generating a sub-query based on a large language model in combination with a multi-modal context of a current iteration step; forming a current state in combination with the sub-query and the multi-modal context, modeling a retrieval enhancement generation task as a Markov decision process, and adaptively selecting an optimal action from a predefined action set in the current state by utilizing a large language model according to a decision strategy; executing a corresponding multi-modal retrieval operation according to the optimal action, fusing the obtained multi-modal information, generating an intermediate answer or a final answer of the sub-query, and updating a multi-modal context by using the intermediate answer; off-line training optimization is carried out on the large language model through imitation learning and a calibration chain, and decision strategies and sub-queries are inferred online through the model after fine adjustment. According to the invention, more efficient and accurate complex query processing is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information retrieval, in particular to a retrieval enhancement generation method and system based on multi-modal fusion and reinforcement learning. BACKGROUND

[0002] In recent years, retrieval-augmented generation (RAG) technology has become an important means to improve the ability of large language models (LLM) to handle knowledge-intensive tasks. Its development mainly reflects the following aspects: (1) RAG based on static text library: represented by DPR and REALM, through single dense text retrieval to enhance LLM input, mainly relying on static knowledge sources such as Wikipedia, laying the foundation for RAG technology, but there are limitations in dealing with complex queries and dynamic knowledge. (2) Evolution of iterative and active text retrieval RAG: represented by DeepRAG, FLARE and Self-RAG, introducing an iterative "thinking-retrieval-reasoning" cycle or a mechanism for LLM to actively trigger retrieval, improving the flexibility and depth of reasoning in handling complex text queries, but still mainly limited to the text modality. (3) Preliminary exploration for visual information processing and application of reinforcement learning: represented by ColPali, which focuses on efficient retrieval directly from document page images, solving the bottleneck of traditional OCR; at the same time, represented by RAG-RL, which attempts to introduce reinforcement learning into text RAG optimization, showing the potential of optimizing complex decision sequences, but multi-modal fusion and intelligent decision-making are still immature.

[0003] Although retrieval-augmented generation technology has made significant progress in improving the performance of large language models, existing RAG systems still have many technical bottlenecks and limitations when facing increasingly complex real-world application scenarios, especially when dealing with queries containing multiple modalities such as text and images. This restricts the further development of its potential, which is manifested in the following three aspects: 1. Serious lack of multi-modal information processing and fusion capabilities: mainstream RAG systems mainly rely on text information retrieval and processing. However, real-world documents (such as PDF reports, web pages, and presentation slides) often contain a large amount of visual elements such as charts, flowcharts, photos, and complex layout designs, which carry rich and sometimes irreplaceable key information by text. Existing text RAG systems cannot effectively extract and understand knowledge from these visual elements.

[0004] 2. The single modality and limitations of the decision mechanism in iterative reasoning: The decision module of existing iterative RAG systems mainly revolves around "whether to retrieve text" or "rely on parameter knowledge". In a multi-modal scenario, the system lacks the intelligent decision-making ability to dynamically determine whether to prioritize text information or visual information based on the current subquery characteristics and context information. The LLM's awareness of its knowledge boundaries is also mainly limited to distinguishing between parameter knowledge and external text knowledge. For questions like "Do I know the content of this chart?" or "Do I need to look at the details of this picture?", the existing mechanism cannot effectively handle them.

[0005] 3. Insufficient complex reasoning and dynamic planning capabilities: Breaking down a complex user query involving multi-modal information into a series of operable subquery sequences that contain both text retrieval and visual retrieval is a significant challenge for existing systems. How to dynamically plan an optimal "retrieval chain" that includes both text retrieval steps and visual retrieval steps and adjust subsequent strategies based on intermediate results, the existing technology lacks mature solutions. SUMMARY

[0006] To solve the technical problems of weak multi-modal information fusion capability and lack of dynamic decision-making and adaptive selection in the reasoning process in the prior art, the present application provides a multi-modal fusion and reinforcement learning collaborative retrieval enhancement generation method and system, which realizes effective fusion of multi-modal information, supports intelligent selection of knowledge sources at each iteration step, and optimizes the decision strategy and knowledge boundary awareness of the large language model through imitation learning and calibration mechanisms.

[0007] To achieve the above purpose, the present application provides the following technical solutions: The present application discloses a multi-modal fusion and reinforcement learning collaborative retrieval enhancement generation method, comprising: receiving a user's input query and combining the multi-modal context of the current iteration step to generate a subquery based on a large language model; Combine the subquery and the multi-modal context to form the current state, model the retrieval enhancement generation task as a Markov decision process, and use a large language model to adaptively select the optimal action from a predefined action set under the current state according to the decision strategy; The action set includes existing knowledge answers for subqueries, text retrieval, visual document retrieval, and termination actions; According to the optimal action, perform the corresponding multi-modal retrieval operation, fuse the obtained multi-modal information, generate the intermediate answer or final answer of the subquery, and update the multi-modal context using the generated intermediate answer each time; Wherein, the large language model is optimized offline through imitation learning and calibration chain, so as to utilize the fine-tuned large language model to make online reasoning and decision strategy and subquery.

[0008] As a further improvement of the above scheme, the multi-modal context includes historical interaction records before the current iteration step and the current accumulated multi-modal information summary; the historical interaction records include all sub-queries of previous iteration steps, actions performed for sub-steps, and corresponding intermediate answers generated; Wherein, in each iteration step, the original query and the multi-modal context are formatted into a text prompt through a preset template, and the text prompt is encoded through a tokenizer of a large language model and then input into the large language model, and the large language model generates a text sequence through autoregressive decoding, which is the sub-query.

[0009] As a further improvement of the above scheme, the selection process of the optimal action specifically includes: Based on the sub-query of the current iteration step and the multi-modal context, a specific prompt for action decision is constructed; the specific prompt is input into the large language model, and the large language model generates a text identifier representing the selected action through autoregressive decoding; then the text identifier is parsed and mapped to a predefined action object; Wherein, in the mapping process, first, the text identifier is exactly string matched with the predefined standard action identifier, if the exact match fails, the text identifier is fuzzy matched with the standard action identifier through keyword extraction or string similarity algorithm; if the confidence of all matching methods is lower than the preset threshold, a preset default action is executed; if no valid action can be parsed, a specific prompt with stronger constraints or clarification requirements is re-initiated to the large language model.

[0010] As a further improvement of the above scheme, when performing the corresponding multi-modal retrieval operation according to the optimal action, if text retrieval is performed, the sub-query of the current iteration step is sent to the text retriever, and the text retriever uses a hybrid retrieval strategy to retrieve text segments related to the sub-query from a pre-constructed text knowledge base, the specific process is: Sparse retrieval of the text knowledge base is performed using the BM25 algorithm to obtain a plurality of initially relevant text segments, thereby generating a sparse retrieval result; at the same time, the sub-query and the document segments in the text knowledge base are encoded into high-dimensional vectors, and then vector indexing is constructed for approximate nearest neighbor search to obtain a plurality of text segments most similar to the sub-query vector, thereby generating a dense retrieval result; the sparse retrieval result and the dense retrieval result are fused through the RRF reordering algorithm to obtain a set of relevant text segments, i.e. the retrieved text information.

[0011] As a further improvement of the above scheme, when performing the corresponding multi-modal retrieval operation according to the optimal action, if visual document retrieval is performed, the sub-query of the current iteration step and the context image retrieved in the previous iteration step are sent to the visual document retriever, and the visual document retriever is called to retrieve the page image related to the sub-query from the pre-constructed image knowledge base. The specific process is as follows: Each page image in the image knowledge base is pre-encoded by a visual encoder into a sequence composed of M image block embedding vectors ; when online retrieval, the sub-query is encoded into a sequence composed of N query token embedding vectors ; then the relevance score of the sub-query and each candidate page image is calculated through a late interaction matching mechanism, and the maximum similarity aggregation strategy is adopted to obtain the top several most relevant page image sets, i.e. the retrieved visual information; wherein the calculation formula of the relevance score is as follows: ; represents the relevance score of the sub-query and the page image ; is the embedding vector formed by encoding the j th image block , ; is the embedding vector formed by encoding the i th query token ; represents the cosine similarity calculation.

[0012] As a further improvement of the above scheme, the generation process of the intermediate answer or the final answer of the sub-query includes: fuse the text information and / or visual information obtained in the current iteration step with the information in the multi-modal context to form a complete context for the current iteration step for answer generation; if the optimal action selected in the current iteration step is a non-termination action, input the sub-query and the complete context into the large language model, and the large language model generates an answer or a relevant information summary to the sub-query through autoregressive decoding as an intermediate answer; if the optimal action selected in the current iteration step is a termination action, or the maximum number of iterations is reached, input the original query and the complete context into the large language model, and the large language model generates an answer to the original query through autoregressive decoding as a final answer; Wherein, before inputting the complete context into the large language model, the semantic relevance of each information segment in the complete context to the current query to be answered is also evaluated using embedding vector cosine similarity, and then the information segments are selected in descending order of semantic relevance scores, and the latest interaction history is preferentially retained until the maximum input length acceptable by the large language model is reached, and the exceeding part is truncated, thereby constructing a prompt containing the complete context and the current query to be answered as input of the large language model.

[0013] As a further improvement of the above scheme, the specific process of offline training and optimization of the large language model includes: Construct a training data set, each sample in the training data set includes an original query and its corresponding standard reference answer; Use a ternary tree search algorithm to automatically explore multiple decision paths from the initial state based on the original query to the final answer that can generate a highly matched standard reference answer for each sample in the training data set; wherein each node of the tree represents a state, and each edge represents an action; Use a pre-defined reward function to evaluate the total reward of each decision path; Select a decision path set with the highest total reward and extract state-optimal action pairs and state-optimal subquery pairs from it to form an imitation learning data set ; wherein, represents the state of the current iteration step, including the original query, the history interaction record and the current accumulated multi-modal context information; represents the optimal action selected according to the decision strategy in state ; represents the next state obtained after executing action , containing updated multi-modal context information and intermediate answers; represents the optimal subquery generated in state , which is a semantic expression of the current subtask and is used to guide subsequent retrieval or generation operations; Use the imitation learning data set to supervise and fine-tune the modules responsible for the decision strategy and subquery generation of the large language model; wherein for the decision strategy generation task, the representation of state is taken as input, and the optimal action is taken as label, and the cross-entropy loss function is used for training; for the subquery generation task, the representation of state is taken as input, and the optimal subquery is taken as the target sequence, and the standard sequence-to-sequence model loss function is used for training.

[0014] As a further improvement of the above scheme, the expression of the predefined reward function is: ; wherein, is the total reward of the decision path ; is the weight coefficient of the answer accuracy; is the final answer generated by the decision path ; is the standard reference answer; is the exact match function, which is 1 if is completely the same as , otherwise 0; is the th action in the decision path ; is the cost weight of the action ; is the cost value of the action .

[0015] As a further improvement of the above scheme, the specific process of offline training and optimization of the large language model further comprises: sampling a plurality of sub-queries and state pairs from the validation set, and using the large language model after the imitation learning phase, i.e. to generate preference data; wherein, for each sub-query and state pair , respectively use to generate subsequent interaction sequences and intermediate answers through three decision paths, and judge the relative advantages and disadvantages of the three decision paths by evaluating the contribution of the intermediate answers generated by each decision path to the standard reference answer; the three decision paths are answering by relying on existing knowledge, performing text retrieval, and performing visual document retrieval; the evaluation standard of the contribution is: calculating three indicators of the intermediate answers generated by each decision path and the standard reference answer, i.e. BLEU score, ROUGE-L and EM score, and then sorting the advantages and disadvantages of the decision paths according to the weighted score after synthesizing the three indicators by weighted average method; construct a preference pair for each sub-query and state pair , and all preference pairs constitute a preference data set ; wherein, is the textual representation of , is the subsequent interaction and answer sequence generated by the decision path evaluated as better; is the subsequent interaction and answer sequence generated by the decision path evaluated as suboptimal; use the preference data set to train the large language model The direct preference optimization training is performed to make the decision strategy tend to select a multimodal decision path that brings better results; wherein, the direct preference optimization loss function is as follows: ; In the formula, is a direct preference optimization loss, is a current strategy to be optimized; is a reference strategy; is an expectation of a sample in a preference data set is an input context; is a preferred response sequence; is a non-preferred response sequence; is a Sigmoid function; is a temperature parameter for controlling the degree of deviation from the reference strategy. The application also discloses a retrieval enhancement generation system cooperating with multimodal fusion and reinforcement learning, which applies the retrieval enhancement generation method cooperating with multimodal fusion and reinforcement learning as described above; the retrieval enhancement generation system comprises:

[0016] a query understanding and iterative decomposition module, configured to receive an original query input by a user and combine a multimodal context of a current iteration step, and generate a subquery based on a large language model; an adaptive retrieval decision module, configured to combine the subquery and the multimodal context to form a current state, model a retrieval enhancement generation task as a Markov decision process, and use the large language model to adaptively select an optimal action from a predefined action set under the current state according to a decision strategy; wherein, the action set comprises an existing knowledge answer, a text retrieval, a visual document retrieval and a termination action for the subquery; an information fusion and generation module, configured to perform a corresponding multimodal retrieval operation according to the optimal action, fuse the obtained multimodal information, generate an intermediate answer or a final answer of the subquery, and update the multimodal context by using the generated intermediate answer each time; a decision optimization module, configured to perform offline training optimization on the large language model through imitation learning and calibration chain, so as to use the fine-tuned large language model to infer the decision strategy and the subquery online. Compared with the prior art, the application has the following beneficial effects:

[0017] 1. Enhanced multimodal information processing and fusion capability: by integrating the visual retrieval capability into the iterative framework, the system can effectively obtain and fuse information from two modalities of text and vision, and significantly improve the understanding and question answering capability for complex documents containing charts, images and the like.

[0018] ​More intelligent and adaptive iterative reasoning strategy: based on the decision framework of Markov decision process and the extended multi-modal imitation learning and calibration chain, the large language model can learn to dynamically select the optimal knowledge source (parameter knowledge, text retrieval or visual retrieval) at each reasoning step according to the current situation, thereby realizing more efficient and accurate complex query processing.

[0019] Enhancing the solving ability of complex problems and the quality of answers: through step-by-step decomposition of problems, multi-modal information retrieval and iterative reasoning, complex problems that require multi-hop reasoning, information integration and cross-modal understanding can be better solved, and more comprehensive, accurate and reliable answers can be generated. Iterative sub-query decomposition and fusion feedback mechanism supports dynamic generation of sub-questions, feeds back the previous information to the next round of retrieval and generation process, and realizes human-like reasoning chain.

[0020] 2. Optimized knowledge boundary awareness: The multi-modal calibration chain of the present application enables the large model not only to judge "whether it knows", but also to judge "whether it is more effective to search text or image for the unknown", thereby reducing unnecessary retrieval and incorrect information dependence. Innovatively introduce "modality preference comparison data" to optimize the decision boundary awareness of LLM under different modalities.

[0021] 3. Automatic generation of high-quality training data: The extended ternary tree search mechanism can automatically generate high-quality training trajectories containing multi-modal decision paths, reducing the dependence on manual annotation of complex reasoning processes, and constructing high-quality imitation learning data. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The flowchart of the multi-modal fusion and reinforcement learning collaborative retrieval enhancement generation method in embodiment 1 of the present application.

[0023] Figure 2 The architecture diagram of the multi-modal fusion and reinforcement learning collaborative retrieval enhancement generation system in embodiment 1 of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0025] Embodiment 1 Please refer to Figure 1 and Figure 2The embodiment provides a retrieval enhancement generation method based on multi-modal fusion and reinforcement learning cooperation. A system corresponding to the method mainly includes the following four core modules: a query understanding and iterative decomposition module, an adaptive retrieval decision module, an information fusion and generation module, and a multi-modal decision optimization module based on imitation learning and calibration. Next, the embodiment will explain the modules and the corresponding methods in detail.

[0026] (1) Query understanding and iterative decomposition module

[0027] This module is used to receive the original query input by the user and generate a sub-query based on a large language model in combination with the multi-modal context of the current iteration step.

[0028] The multi-modal context includes historical interaction records before the current iteration step and a multi-modal information summary that has been accumulated so far The historical interaction records include all sub-queries of previous iteration steps, actions performed for the sub-steps, and corresponding intermediate answers generated. Wherein, in each iteration step, the original query and the multi-modal context are formatted into a complete text prompt through a preset template The prompt template is designed as: "The original query is: [fill in ]. In combination with the following reasoning steps and known information that have been performed: [fill in the formatted and ]. What is the simplest, most explicit and actionable sub-question that should be asked next to answer the original query? Please give the sub-question directly."

[0029] In this embodiment, the core of the iterative decomposition query is a large language model based on Llama-3.1-8B-Instruct fine-tuning. The generation of sub-queries decomposes complex problems into a series of simple problems that are easier to answer through retrieval or parameter knowledge. The query understanding and iterative decomposition module receives the original query input by the user In each iteration step t a next atomized and actionable sub-query is generated according to the current multi-modal context and the original query The process is iterative, and the generation of the sub-query will consider and the corresponding processing results, which can be expressed as follows: ; Wherein, representing the decomposition function of the LLM, are LLM parameters.

[0030] The Llama-3.1-8B-Instruct model mentioned above can be loaded through the Transformers library of HuggingFace, which includes loading its corresponding tokenizer and model weights, which are obtained after fine-tuning training by the decision optimization module. The text prompt is encoded by the tokenizer of the large language model as input, and then input into the large language model. The large language model generates a text sequence, i.e. a subquery During the generation process, the temperature is set to 0.7, and the sampling parameter is 0.9 to control the creativity and concentration of the generated subquery. After generation, the output is post-processed to extract the pure subquery text.

[0031] This method enables the system to gradually decompose complex problems, reducing information overload in a single processing and improving the orderliness and controllability of problem solving.

[0032] In addition, this method also has context dependence: the generation depth of the subquery depends on the previous reasoning steps and the acquired multi-modal information, ensuring the coherence and goal-oriented nature of the reasoning chain. It avoids generating subqueries that are irrelevant or repetitive to the current reasoning goal, improving the efficiency and relevance of the entire reasoning process.

[0033] (2) Adaptive retrieval decision module

[0034] This module is used to combine the subquery and the multi-modal context to form the current state, model the retrieval-enhanced generation task as a Markov decision process, and use the large language model to adaptively select the optimal action from a predefined action set according to the decision strategy in the current state; the action set includes existing knowledge answers for the subquery, text retrieval, visual document retrieval, and termination actions.

[0035] In this embodiment, the selection process of the optimal action specifically includes: the subquery based on the current iteration step and the multi-modal context , a specific prompt for action decision is constructed , which explicitly lists the current subquery, key information of the context, and provides predefined and clear multi-modal action options for the LLM to choose. The prompt template is designed as: "The current subproblem to be solved is: [fill in ]. According to the context information currently mastered: [fill in Please select the most appropriate action from the following to answer the current sub-question (please strictly return the English identifier of the selected action): 1. (Directly answer using existing knowledge); 2. (retrieve relevant text); 3. (retrieve relevant images or visual documents); 4. (There is enough information to end the search and generate the final answer.) Your decision is: ".

[0036] The specific prompt is input into the large language model, which generates a text identifier representing the selected action through autoregressive decoding, such as " "; Then parse the text identifier and map it to a predefined action object For example, if LLM outputs " ",but Set to perform visual retrieval actions and carry parameters .

[0037] In the mapping process, the text identifier is first matched with the predefined standard action identifier by exact string matching. If the exact match fails, the text identifier is matched by keyword extraction (for example, if "text" or "文字" is identified, the text identifier is matched with the predefined standard action identifier). ; recognizes "visual", "image", "images", or "charts" and prefers ) or string similarity algorithms such as Levenshtein distance) to perform fuzzy matching between the text identifier and the standard action identifier; if the confidence of all matching methods is lower than the preset threshold (set to 0.7), the preset default action (conservatively select ), if no valid action can be parsed, a new prompt with stronger constraints or clarification requirements is issued to the large language model. The final action is Will be used to guide subsequent information retrieval or generation steps. If the action requires parameters (such as a subquery for retrieval), Passed as a parameter.

[0038] Multimodal state representation and action space: The adaptive retrieval decision module receives the query understanding and iterative decomposition modules , obtained from the information fusion and generation module To constitute the state ; Adaptive retrieval decision module based on the current state and its internally learned decision strategy to select actions , according to the decision action ,Will The decision logic is learned by the decision optimization module, aiming to maximize the expected cumulative reward.

[0039] The adaptive retrieval decision module formalizes the RAG process into a Markov decision process MDP, which is expressed as follows: ; Where, S Represents the state space, which represents the complete information environment of the system at each decision moment; For the extended multimodal action space, in each state , LLM (as a strategy Executor) selects an action from a predefined set of actions The set of predefined actions contains knowledge answers that depend on LLM parameters. of ,right Perform text retrieval ,right Perform visual document retrieval , marks the termination of iteration and is ready to generate the final answer The multimodal action space enables the system to flexibly select the most appropriate knowledge source based on current information needs, breaking through the limitations of the traditional RAG system's single retrieval modality. P Represents the state probability transfer function, which describes the state S t Next action a t After that, the system moves to the new state S t+1 The probability distribution of R represents the reward function, which is used to evaluate the immediate benefit of performing an action in a specific state; gamma Represents the discount factor, which is a constant between 0 and 1 and is used to adjust the value weight of future rewards in current decisions. gamma A value close to 1 indicates a focus on long-term gains and a willingness to accept immediate costs for future returns; gamma A value close to 0 indicates a greater focus on immediate rewards and a tendency toward short-term, efficient behavior.

[0040] Strategic decision making: Action selection is based on optimizing the strategy learned by the LLM in the decision optimization module:

[0041] in In state Next, select the action. The parameters are The policy network is represented in the state The probability distribution of selecting each action is denoted by the policy π. The policy aims to maximize the expected cumulative reward, i.e., the state value function : ; where represents the expectation under the policy π. is a discount factor, ; is the immediate reward obtained at future time step t+k+1.

[0042] The optimization goal is to find the optimal policy such that holds for all states s and all policies π. The policy trained by reinforcement learning makes the decision-making process more intelligent and adaptive, enabling dynamic adjustment of information acquisition methods according to specific problem scenarios, thereby improving the accuracy and necessity of retrieval and avoiding unnecessary retrieval overhead.

[0043] As an agent, LLM can adaptively select from a multi-modal action space that includes parameter knowledge utilization, text information retrieval, and visual document information retrieval based on page images (using the late interaction multi-vector matching principle) at each decision step according to the current state, and iteratively perform query understanding and decomposition.

[0044] (3) Information fusion and generation module

[0045] This module is used to perform corresponding multi-modal retrieval operations according to the optimal action, fuse the obtained multi-modal information, generate intermediate answers or final answers of the subquery, and update the multi-modal context using the intermediate answer generated each time.

[0046] Among them, text retrieval and visual document retrieval can be retrieved in parallel. If text retrieval is performed , the subquery of the current iteration step is sent to the text retriever, which uses a hybrid retrieval strategy to retrieve text segments related to the subquery from the pre-constructed text knowledge base . The specific process is as follows: Use the electrostatic BM25 algorithm (implemented through the Pyserini library) to perform sparse retrieval on the text knowledge base (a collection of Wikipedia paragraphs with established inverted indexes), obtain the top several (Top-N1) preliminary relevant text segments, and generate sparse retrieval results; at the same time, load the intfloat / e5-large-v2 model from HuggingFace Hub to retrieve text segments related to the subquery The document fragments in the text knowledge base are encoded into high-dimensional vectors, and then the vector index is constructed using the FAISS library for approximate nearest neighbor search to obtain several (Top-N2) text fragments that are most similar to the subquery vector, thereby generating dense retrieval results; finally, the sparse retrieval results and the dense retrieval results can be fused through the RRF (Reciprocal Rank Fusion) re-ranking algorithm to obtain the final Top-K text The collection of related text fragments is the retrieved text information.

[0047] If you perform visual document retrieval , then the subquery of the current iteration step and the context image retrieved in the previous iteration step Send to the visual document retriever, call the visual document retriever to retrieve the image from the pre-built image knowledge base Search and subquery Related page images, the specific process is: Access to a pre-processed image library, each page image in the library The google / siglip-so400m-patch14-384 visual model and its corresponding image processor AutoImageProcessor have been divided into M image blocks (paches), each image block is encoded as an embedding vector , these embedding vectors are stored using FAISS and form a sequence ; When searching online, you can use the intfloat / e5-large-v2 model in text search to convert subqueries Encoded by N A sequence of query token embedding vectors ; Then, the relevance score between the subquery and each candidate page image is calculated through the post-interactive matching mechanism, and the maximum similarity aggregation (MaxSim) strategy is adopted to obtain the top three most relevant page image sets, i.e., the retrieved visual information; wherein, the calculation formula of the relevance score is: ; Represents a subquery With page image 's relevance score; For the j Image blocks The embedded vector formed by encoding, ; For the ian embedding vector formed by encoding the query token, ; denotes a cosine similarity calculation.

[0048] Multi-modal retrieval enables the system to directly utilize visual elements in the document (such as charts, illustrations), obtaining deep information that pure text retrieval cannot obtain, significantly enhancing the understanding ability of multi-modal documents.

[0049] The generation process of the intermediate answer or the final answer of the subquery includes: the text information obtained in the current iteration step and / or visual information (for each page image in the visual information, a concise text description can be generated for the image using the LLaVA-1.5 model, and the text description output by LLaVA will be normalized and uniformly embedded in the embedding space), and the information accumulated in the previous multi-modal context is input into a fusion layer with cross-modal attention mechanism (the fusion layer adopts Transformer structure, and Cross-Modality Attention is introduced in the encoding stage, and the visual description embedding and the text information are input together), forming the complete context for answer generation in the current iteration step , that is: ; wherein, is a multi-modal information fusion function; is the representation after processing the original visual retrieval results (such as text description or compatible embedding). Information fusion ensures that information from different sources and different modalities can be uniformly understood and utilized by the LLM, laying a foundation for generating high-quality and information-rich answers and improving the problem of information fragmentation.

[0050] Answer generation: based on the subquery and the complete context , the LLM is used to generate the intermediate answer or the final answer . In this embodiment, the answer generation can use the GPT-4o model called through the OpenAI API. Before inputting the complete context into the GPT-4o model, the semantic relevance of each information segment in the complete context to the current query to be answered is also evaluated using the embedding vector cosine similarity, and then the information segments are selected in descending order of semantic relevance scores, and the latest interaction history is preferentially retained until the maximum input length acceptable by the large language model is reached, and the part exceeding the maximum input length is truncated, thereby constructing a prompt , as input to the GPT-4o model. The template is designed as "Please answer the question: [Fill in the formatted ] with the answer: [Fill in or ]". After receiving this prompt, the GPT-4o model generates the corresponding intermediate answer or final answer through self-recursive decoding.

[0051] In each iteration step t , if the optimal action selected by the adaptive retrieval decision module is not the termination action (that is, the iteration is not over), the subquery and the complete context are input into the large language model, and the large language model generates the answer to the subquery or the relevant information summary as the intermediate answer through self-recursive decoding. This process can be represented as: ; where represents the function of the LLM executing the intermediate answer generation; will be used to update the context for the next iteration step.

[0052] In each iteration step t , if the optimal action selected by the adaptive retrieval decision module is the termination action , or the maximum number of iterations is reached, the generation of the final answer is triggered. At this time, the original query and the final complete context accumulated at the end of the iteration are input into the large language model, and the large language model generates the answer to the original query as the final answer through self-recursive decoding. This process can be represented as: ; where represents the function of the LLM executing the final answer generation.

[0053] By making full use of the fused multi-modal context, the generated answer is more factual, relevant and deep, effectively reducing the "illusion" phenomenon of the model.

[0054] (4) Decision optimization module

[0055] The module is used for offline training optimization of a large language model through imitation learning and a calibration chain, so as to utilize the large language model after fine-tuning to online infer decision strategies and sub-queries.

[0056] The core task of the decision optimization module is to train and optimize the decision strategies of the LLM , so that it can perform efficient and accurate multi-modal iterative inference. The specific process of offline training optimization of the large language model includes: Constructing a training data set, each sample in the training data set including an original query and its corresponding standard reference answer.

[0057] Automated multi-modal trajectory data synthesis: using a ternary tree search algorithm (branches correspond to core actions such as parameter knowledge answering, text retrieval, and visual document retrieval) to automatically explore multiple trajectories for each sample in the training data set from the initial state based on the original query to the final answer that can generate a high degree of match with the standard reference answer , i.e., a decision path; where each node of the tree represents a state and each edge represents an action.

[0058] A predefined reward function is used to evaluate the total reward of each decision path; the expression of the predefined reward function is: ; In the formula, is the total reward of the decision path ; is the weight coefficient of answer accuracy, generally taking 0.7; is the final answer generated by the decision path ; is the standard reference answer (gold standard answer); is the precise matching function, which is 1 if is completely the same as , otherwise it is 0; is the th action in the decision path ; is the cost weight of action , taking 0.3; is the cost value of action ; is set to 0.002 because it mainly consumes the inference ability of the LLM itself; is set to 0.1, reflecting the overhead of text encoding, index query, and result processing; is set to 0.15, which is set to the highest because it involves image encoding and complex visual feature matching; is set to 0.

[0059] select the decision path set with the highest total reward, and extract state-optimal action pairs from it and state-optimal subquery pairs to form an imitation learning dataset ; wherein, represents the state of the current iteration step, including the original query, historical interaction records and the current accumulated multi-modal context information; represents the optimal action selected according to the decision strategy under the state ; represents the next state obtained after executing the action , containing updated multi-modal context information and intermediate answers; represents the optimal subquery generated under the state , which is the semantic expression of the current subtask, used to guide subsequent retrieval or generation operations. This method realizes the automatic generation of high-quality training data, reduces the dependence on expensive manual annotation data, and improves the scalability of training.

[0060] use the imitation learning dataset to supervise and fine-tune the modules responsible for the decision strategy and subquery generation of the large language model.

[0061] Among them, for the decision strategy generation task, the representation of state is taken as input, and the optimal action is taken as label, and cross-entropy loss function is used for training, and the loss function is as follows: ; In the formula, is the loss function of the imitation learning strategy part; is the state-action pair extracted from the optimal trajectory; is the probability of the model selecting the optimal action under the state .

[0062] For the subquery generation task, the representation of state is taken as input, and the optimal subquery is taken as the target sequence, and the standard sequence-to-sequence model loss function is used for training. The two tasks are optimized simultaneously using a multi-task learning framework. This stage of training uses HuggingFace's Trainer API.

[0063] To further improve the ability of LLM to distinguish the application boundary of different knowledge modalities, a number of subqueries and state pairs are sampled from the validation set, and a large language model after the imitation learning stage is used as Generate preference data ; wherein the validation set is a part of the training data set reserved in advance, and the model training process does not participate in the imitation learning stage, and is only used for subsequent preference data generation and evaluation to ensure the independence and publicity of the validation data. For each subquery and state pair , the subsequent interaction sequence and intermediate answer generated by forcing to explore three decision paths respectively, the contribution of the intermediate answer generated by each decision path to the standard reference answer is evaluated to judge the relative advantages and disadvantages of the three decision paths; the three decision paths are respectively to rely on existing knowledge to answer, to execute text retrieval and to execute visual document retrieval, and each path only uses a single type of action when generating an intermediate answer, without mixing different retrieval methods to ensure the comparability and independence between paths; the evaluation standard of contribution is: calculate the three indicators of the intermediate answer generated by each decision path and the standard reference answer, that is, the BLEU score, ROUGE-L and EM score, and then use the weighted average method to integrate the three indicators, and then sort the advantages and disadvantages of the decision paths according to the weighted score; For each subquery and state pair , a preference pair is constructed , and all preference pairs constitute a preference data set ; wherein is the textual representation of , is the subsequent interaction and answer sequence generated by the decision path evaluated as the better one; is the subsequent interaction and answer sequence generated by the decision path evaluated as the less optimal one; The preference data set is used to train the large language model using direct preference optimization (DPO) to make its decision strategy tend to choose the multi-modal decision path that brings better results; wherein the direct preference optimization loss function is as follows: ; In the formula, is the direct preference optimization loss, is the current strategy to be optimized; is the reference strategy; is the expectation of the sample in the preference data set ; is the input context; is the winning response sequence; is the losing response sequence; is a Sigmoid function; is a temperature parameter, used to control the degree of deviation from the reference policy.

[0064] Direct preference optimization training can be implemented using the DPOTrainer class in the Transformer Reinforcement Learning (TRL) library. When configuring the DPOTrainer, the reference model is an initial copy of the model to be trained (with its parameters fixed before the start of DPO training). All input contexts and response sequences will be uniformly formatted to fit the DPO input format. In the loss function of DPO, the temperature parameter is set to 0.1. Through DPO training, the decision logic of the LLM is adjusted to prefer the selection of multi-modal decision paths that can lead to better results. After training is complete, the final optimized LLM is obtained for online inference. In this embodiment, LlaMA2-7B can be used, which is adapted to the multi-modal decision generation task through imitation learning and DPO optimization in the offline training phase.

[0065] Through direct preference optimization training, the accuracy of the LLM's understanding of its own knowledge boundaries in different modalities is significantly improved, enabling it to more intelligently determine when to rely on internal knowledge, and when and how to utilize external knowledge (text or visual), thereby optimizing the necessity and effectiveness of retrieval and further improving the reliability of the answer.

[0066] In summary, after receiving the user's original query, the RAG system of the present application first generates the first sub-query by the query understanding and iterative decomposition module. Then, the adaptive retrieval decision module selects an optimal multi-modal action for the sub-query based on the current state. The information fusion and generation module is responsible for executing the action and fusing the retrieved information to generate an intermediate answer to the sub-query, while updating the multi-modal context. This "sub-query generation-multi-modal decision-retrieval-information fusion-intermediate answer generation" process is iterated. At each step, the decision logic of the LLM is shaped by the offline-trained decision optimization module. When the adaptive retrieval decision module determines that sufficient information has been collected or the preset termination condition is reached, the information fusion and generation module generates the final answer to the original query based on the final complete context. Through this iterative, multi-modal perception processing flow, the present application can better understand user intent and more comprehensively obtain the required information, thereby generating a higher quality response.

[0067] To verify the effectiveness of the method of the present application, the following experiments are provided in this embodiment: I. Experimental setup Experimental task: Multi-round question-answering retrieval augmented generation task, covering complex queries in mixed text and image scenarios, such as document understanding, table question-answering, visual reasoning, etc.

[0068] Dataset source: Construct a test set containing 1000 multi-modal original queries (text and related images) and artificially annotated answers; all data contain explicit reference answers for accuracy evaluation.

[0069] Evaluation indicators: Accuracy indicators of generated answers: BLEU, ROUGE-L, EM (Exact Match); multi-modal decision rationality indicators: action selection and retrieval path matching degree.

[0070] Comparison method: Baseline 1: Only based on LLM question-answering, without introducing any retrieval module; Baseline 2: Traditional RAG (text retrieval augmentation); Baseline 3: Multi-modal RAG (image-text retrieval augmentation), but without introducing reinforcement learning and path selection; The method of the present application: combination of reinforcement learning modeling, subquery generation, adaptive action selection, imitation learning and DPO optimization.

[0071] II. Experimental results

[0072] As shown in Table 1, the action rationality rate represents the proportion of the model's selected actions consistent with the true optimal path, reflecting the model's path decision-making ability.

[0073] Table 1: Comparison results of evaluation indicators of each method ; Note: In Table 1, the upward arrow "↑" indicates that the higher the result of this indicator, the better, and the downward arrow "↓" indicates that the lower the result of this indicator, the better.

[0074] III. Experimental conclusion The experimental results show that compared with existing methods, the method of the present application has achieved significant improvement in accuracy indicators (BLEU, ROUGE-L, EM), especially in multi-modal decision path selection and execution efficiency, showing higher intelligence and robustness. In particular, by introducing the reinforcement learning framework and DPO optimization training, the large language model can more effectively select the optimal action path in different states, thereby improving the overall performance of the retrieval augmented generation system. Therefore, the method of the present application can significantly improve the retrieval generation quality of large language models in complex, multi-modal scenarios, and has significant technical effects and promotional value.

[0075] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A retrieval enhancement generation method based on multimodal fusion and reinforcement learning, characterized by: include: Receive the original query input by the user and combine it with the multimodal context of the current iteration step to generate sub-queries based on the large language model; Combining the subquery and the multimodal context to form a current state, modeling the retrieval-enhanced generation task as a Markov decision process, and utilizing a large language model to adaptively select an optimal action from a predefined set of actions in the current state according to a decision strategy; the action set includes prior knowledge answers to the subquery, text retrieval, visual document retrieval, and a termination action; Executing a corresponding multimodal retrieval operation according to the optimal action, fusing the acquired multimodal information, generating an intermediate answer or a final answer to the subquery, and updating the multimodal context using each generated intermediate answer; Among them, the large language model is trained and optimized offline through imitation learning and calibration chain, so that the fine-tuned large language model can be used to infer decision strategies and subqueries online.

2. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 1 is characterized in that: The multimodal context includes historical interaction records before the current iteration step and the currently accumulated multimodal information summary; the historical interaction records include all sub-queries of the previous iteration step, actions performed for the sub-steps, and corresponding intermediate answers generated; In each iterative step, the original query and the multimodal context are formatted into a text prompt using a preset template, and the text prompt is used as input, encoded by the word segmenter of the large language model, and then input into the large language model. The large language model generates a text sequence, namely the subquery, through autoregressive decoding.

3. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 1 is characterized in that: The optimal action selection process specifically includes: Based on the subquery and multimodal context of the current iteration step, a specific prompt for action decision is constructed; the specific prompt is input into the large language model, and the large language model generates a text identifier representing the selected action through autoregressive decoding; the text identifier is then parsed and mapped to a predefined action object; In the mapping process, the text identifier is first precisely matched with the predefined standard action identifier. If the precise match fails, the text identifier is fuzzy matched with the standard action identifier through keyword extraction or string similarity algorithm. If the confidence of all matching methods is lower than the preset threshold, the preset default action is executed. If no valid action can be parsed, a specific prompt with stronger constraints or clarification requirements is re-initiated to the large language model.

4. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 1 is characterized in that: When executing the corresponding multimodal retrieval operation according to the optimal action, if a text retrieval is performed, the subquery of the current iteration step is sent to the text retriever. The text retriever uses a hybrid retrieval strategy to retrieve text fragments related to the subquery from the pre-built text knowledge base. The specific process is as follows: The BM25 algorithm is used to perform sparse search on the text knowledge base to obtain the first several preliminarily relevant text fragments, thereby generating sparse search results; at the same time, the subquery and the document fragments in the text knowledge base are encoded into high-dimensional vectors, and then a vector index is constructed to perform approximate nearest neighbor search to obtain several text fragments whose vectors are most similar to the subquery vector, thereby generating dense search results; the sparse search results and the dense search results are fused through the RRF reranking algorithm to obtain a set of relevant text fragments, namely the retrieved text information.

5. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 4 is characterized in that: When executing the corresponding multimodal retrieval operation according to the optimal action, if visual document retrieval is performed, the subquery of the current iteration step and the context image retrieved in the previous iteration step are sent to the visual document retriever, and the visual document retriever is called to retrieve page images related to the subquery from the pre-built image knowledge base. The specific process is as follows: Use the visual encoder to pre-encode each page image in the image knowledge base into M A sequence of image patch embedding vectors ; When searching online, encode the subquery as N A sequence of query token embedding vectors ; Then, the relevance score between the subquery and each candidate page image is calculated through a post-interactive matching mechanism, and a maximum similarity aggregation strategy is adopted to obtain the first several most relevant page image sets, i.e., the retrieved visual information. The calculation formula for the relevance score is: ； Represents a subquery With page image 's relevance score; For the j Image blocks The embedded vector formed by encoding, ; For the i The embedding vector formed by encoding the query token, ; Represents cosine similarity calculation.

6. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 5 is characterized in that: The process of generating the intermediate answer or final answer of the subquery includes: Fusing the textual information and / or visual information obtained in the current iteration step with the information in the multimodal context to form a complete context for answer generation in the current iteration step; If the optimal action selected in the current iteration step is not a terminal action, the subquery and the complete context are input into the large language model, and the large language model generates an answer to the subquery or a summary of related information as an intermediate answer through autoregressive decoding; If the optimal action selected in the current iteration step is the termination action, or the preset maximum number of iterations is reached, the original query and the complete context are input into the large language model, and the large language model generates an answer to the original query, i.e., the final answer, through autoregressive decoding; Before the complete context is input into the large language model, the semantic relevance of each information fragment in the complete context with the current query to be answered is evaluated using the cosine similarity of the embedded vectors. Then, information fragments are selected from high to low according to the semantic relevance score, and the latest interaction history is retained first until the maximum input length acceptable to the large language model is reached. The excess part is truncated, thereby constructing a prompt containing the complete context and the current query to be answered, which is used as the input of the large language model.

7. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 1 is characterized in that: The specific process of offline training and optimization of the large language model includes: Construct a training dataset, where each sample in the training dataset includes the original query and its corresponding standard reference answer; A ternary tree search algorithm is used to automatically explore multiple decision paths for each sample in the training dataset, from the initial state based on the original query to the final answer that can generate a high match with the standard reference answer. Each node of the tree represents a state, and each edge represents an action. Use a predefined reward function to evaluate the total reward of each decision path; Select the set of decision paths with the highest total reward and extract the optimal state-action pair from them and state-optimal subquery pair , constitutes the imitation learning dataset ;in, Represents the state of the current iteration step, including the original query, historical interaction records, and currently accumulated multimodal context information; Indicates that the status The optimal action selected according to the decision strategy; Indicates that an action is being performed The next state obtained after , contains the updated multimodal context information and the intermediate answer; Indicates that the status The optimal subquery generated is used as the semantic expression of the current subtask to guide subsequent retrieval or generation operations; Using Imitation Learning Datasets The modules responsible for decision strategy and subquery generation of the large language model are fine-tuned by supervision; for the decision strategy generation task, the state The representation of is taken as input, the optimal action As the label, the cross entropy loss function is used for training; for the subquery generation task, the state The representation of is taken as input and the optimal subquery is As the target sequence, the standard sequence-to-sequence model loss function is used for training.

8. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 7 is characterized in that: The expression of the predefined reward function is: ； Where, Decision path Total rewards; is the weight coefficient of the answer accuracy; Decision path The final answer produced; For the standard reference answer; is an exact matching function, if and If they are exactly the same, it is 1, otherwise it is 0; Decision path The Actions For action Cost weights; For action cost value.

9. The retrieval enhancement generation method based on multimodal fusion and reinforcement learning collaboration according to claim 7 is characterized in that: The specific process of offline training and optimization of the large language model also includes: Several subqueries and state pairs are sampled from the validation set, and the large language model after the imitation learning phase is used. Generate preference data; where, for each subquery and state pair , respectively using The three decision paths generate subsequent interaction sequences and intermediate answers. The relative merits of the three decision paths are determined by evaluating their contribution to the standard reference answer. The three decision paths are: answering based on prior knowledge, performing text retrieval, and performing visual document retrieval. The contribution evaluation criteria are: calculating three metrics, namely, the BLEU score, ROUGE-L, and EM score, comparing the intermediate answer generated by each decision path to the standard reference answer. After using a weighted average method to combine the three metrics, the decision paths are ranked according to their weighted scores. For each subquery and state pair Construct a preference pair , all preference pairs constitute the preference data set ;in, for The textual representation of the sequence of subsequent interactions and answers generated for the decision path that was evaluated as superior; the subsequent sequences of interactions and answers generated for decisions that passed through a decision path that was assessed as suboptimal; Use Preference Dataset For large language models Direct preference optimization training is performed to make its decision strategy tend to choose the multimodal decision path that brings better results; the direct preference optimization loss function is as follows: ； Where, Optimizing loss for direct preference, is the strategy to be optimized; For reference strategy; For the preference dataset Expectations of the sample; To input context; is the preferred response sequence; is a non-preferred response sequence; is the Sigmoid function; is the temperature parameter used to control the degree of deviation from the reference strategy.

10. A retrieval enhancement generation system based on multimodal fusion and reinforcement learning, characterized by: Applying the retrieval enhancement generation method of multimodal fusion and reinforcement learning collaboration as described in any one of claims 1 to 9; The retrieval enhancement generation system includes: The query understanding and iterative decomposition module receives the original query input by the user and combines it with the multimodal context of the current iteration step to generate subqueries based on the large language model; an adaptive retrieval decision module, configured to combine the subquery and the multimodal context to form a current state, model the retrieval enhancement generation task as a Markov decision process, and adaptively select the optimal action from a predefined set of actions in the current state based on a decision strategy using a large language model; wherein the action set includes prior knowledge answers to the subquery, text retrieval, visual document retrieval, and a termination action; An information fusion and generation module, configured to perform corresponding multimodal retrieval operations according to the optimal action, fuse the acquired multimodal information, generate an intermediate answer or a final answer to the subquery, and update the multimodal context using each generated intermediate answer; The decision optimization module is used to optimize the offline training of the large language model through imitation learning and calibration chain, so as to use the fine-tuned large language model to infer decision strategies and subqueries online.

Images