Multimodal Agent Enhanced Retrieval Algorithm

Abstract

This invention discloses a multimodal agent-enhanced retrieval algorithm, belonging to the field of enhanced retrieval algorithm technology. The invention includes a multimodal text conversion unit, a vector database construction unit, and a multimodal knowledge graph (MMKG) construction unit. It enhances cross-modal reasoning capabilities, solving the modality disconnect problem in existing technologies: establishing visual-text associations through MMKG. It improves the efficiency of multi-source knowledge synthesis by employing plug-in parallel retrieval. Through consistency verification of the decision-making agent and expert refinement, it reduces the answer illusion rate, i.e., the proportion that conflicts with facts; MMKG embeds visual data storage paths, and web retrieval provides URL tracing, allowing users to trace the source of each fact in the answer, thus solving the black-box reasoning problem in existing RAGs.

Description

Technical Field

[0001] This invention relates to the field of enhanced retrieval algorithm technology, specifically to a multimodal agent enhanced retrieval algorithm. Background Technology

[0002] With the explosive growth of data scale and the increase in modal diversity, retrieval augmentation (RAG) technology has become a core solution to compensate for the knowledge lag of large language models and reduce illusions, and is widely used in fields such as scientific question answering, crisis event analysis, and medical consultation. Traditional RAG systems mainly focus on a single modality and a single data source, but in practical applications, user queries often require collaborative reasoning across heterogeneous data. This places higher demands on the multimodal processing, multi-source retrieval collaboration, and cross-modal knowledge synthesis capabilities of RAG systems. Currently, multimodal RAG and multi-agent RAG have become research hotspots, but existing solutions still struggle to balance "retrieval comprehensiveness" and "reasoning accuracy." A modular and collaborative architecture is urgently needed to solve the challenges of heterogeneous data reasoning and multi-source knowledge synthesis. Therefore, it is necessary to analyze multimodal agent-enhanced retrieval algorithms.

[0003] Existing technologies, such as the sentiment analysis method and system based on multimodal enhanced retrieval generation technology disclosed in invention application patent CN120524444B, belong to the fields of sentiment analysis and natural language processing. The method includes: constructing an external multimodal vector database for sentiment analysis tasks; converting user input data to obtain query vectors; retrieving target feature vectors from the external multimodal vector database using a vector retrieval algorithm based on the query vectors corresponding to the user input data; constructing a sentiment analysis prompt template; extracting other field data corresponding to the target feature vectors from the external multimodal vector database as retrieval results based on the target feature vectors; embedding the user input data and the retrieval results from the external multimodal vector database into the prompt template; and calling a large language model to generate sentiment analysis results.

[0004] Existing technologies for multimodal agent-enhanced retrieval algorithms can meet basic requirements, but they also have some potential defects and challenges, which are reflected in the following aspects: First, the problems of single-agent single-modal RAG systems: limited modality coverage, inability to handle cross-modal queries, such as explaining the cause of a fault based on a circuit diagram in an image and a text manual, can only respond to a single modal input, the knowledge source is singular, relying on a single data source such as a text vector library, and cannot integrate entity relationships of graph data and real-time information of the network, resulting in incomplete information under complex queries.

[0005] II. Problems with Single-Agent Multimodal RAG Systems: Weak Cross-Modal Associations: Although multimodal data is integrated, the system only unifies modalities through simple methods such as image-to-text conversion, failing to establish fine-grained cross-modal associations, such as the correspondence between specific regions in an image and certain terms in text. This leads to a disconnect in modal information during inference. Low Retrieval Efficiency: A single retrieval module processes multimodal data, making it impossible to optimize retrieval strategies for different data types. For example, text requires fine-grained vector retrieval, while graph data requires multiple traversals, resulting in high retrieval latency and low accuracy.

[0006] III. Problems with Multi-Agent Single-Modal RAG Systems: Difficulty in Modal Expansion: Multi-agent division of labor is limited to a single modality, such as text design. Adding visual, network, or other modalities requires refactoring the agent interaction logic, resulting in poor flexibility. Lack of Cross-Modal Reasoning: Unable to handle cross-modal collaborative reasoning scenarios such as text query + image evidence, graph relationships + real-time network information, limiting the scope of application. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal agent-enhanced retrieval algorithm that solves the problems existing in the background art.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a multimodal agent-enhanced retrieval algorithm, including a user multimodal query input module, a multi-source retrieval agent module, and a decision agent module.

[0009] User multimodal query input module: Multimodal knowledge preprocessing. Before the framework runs, multimodal knowledge preprocessing needs to be completed to provide a data foundation for retrieval. This includes a multimodal text conversion unit, a vector database construction unit, and a multimodal knowledge graph (MMKG) construction unit.

[0010] Multi-source retrieval agent module: Three sub-components work in parallel, each handling different types of data sources.

[0011] Decision-making agent module: Integrates and refines answers from multiple sources.

[0012] Furthermore, the specific analysis method of the multimodal text conversion unit is as follows: Multimodal text conversion: Using the BLIP-2 framework, image features are extracted through a visual encoder, and visual information is converted into structured text descriptions by combining a cross-modal alignment module, and a context refinement mechanism is introduced.

[0013] Furthermore, the specific analysis method of the vector database construction unit is as follows: The original text T and the refined visual structured text description Tv are merged into a multimodal text library, and the multimodal text aggregate is encoded into... A dimensional vector, stored in a vector database, is used for vector retrieval by intelligent agents.

[0014] Furthermore, the specific calculation formula for the multimodal text library is as follows: .

[0015] Furthermore, the specific analysis method for the multimodal knowledge graph (MMKG) construction unit is as follows: MMKG construction: Based on the multimodal text aggregate Tm, entity relation extraction is performed using the LightRAG framework, and the function... Extract Entities relational triples By combining two-level reasoning enhancement, an MMKG is constructed. It also embeds a visual data storage path to support cross-modal tracing.

[0016] Furthermore, the three sub-components include a vector retrieval agent, a graph retrieval agent, and a network retrieval agent.

[0017] Furthermore, the specific analysis method of the vector retrieval agent is as follows: Input: Subtask The text form.

[0018] Step 1: Encode using a text encoder query vector .

[0019] Step 2: Calculation With all document vectors in the vector database cosine similarity .

[0020] Step 3: Retrieve documents with the top-k similarity Input a lightweight LLM to generate an answer. Constraining the decoding parameters (top-p=1.0, temperature=0) ensures deterministic output.

[0021] Furthermore, the graph retrieval agent's specific analysis method is as follows: Input: Subtasks in text form .

[0022] Step 1: Large Model Semantic Generation Local keywords and global keywords.

[0023] Step 2: Hybrid Graph-Vector Matching: Align local keywords with MMKG entity attributes, and match global keywords with MMKG relationship patterns.

[0024] Step 3: Subgraph Expansion: Retrieve relevant triples to construct a subgraph This extends to the 1-hop neighbors (Nu, Ne) of the node and edge to ensure structural integrity.

[0025] Furthermore, the specific analysis method of the network retrieval agent is as follows: Input: Subtask The text form.

[0026] Step 1: Call the bochaAPI or other search tools.

[0027] Step 2: Obtain the structured results returned by the API Includes title, abstract, URL, and ranking.

[0028] Step 3: Filter redundant results and extract key information to generate the answer. .

[0029] Furthermore, the specific analysis method for integrating and refining multi-source answers is as follows: the voting of the large model for filtering user information is verified through experiments, and the threshold for deduplication of information is determined by the proportion and embedding model. Multiple lightweight models vote on the input information to determine whether the retrieved information is relevant to the question. When the proportion of relevant voting results reaches the threshold, the information is retained.

[0030] The beneficial effects of this invention are as follows: 1. Enhanced cross-modal reasoning capabilities, solving the modal disconnect problem in existing technologies: establishing visual-textual associations through MMKG. 2. Improved efficiency in multi-source knowledge synthesis, employing plug-in parallel retrieval. 3. Reduced answer illusion rate (i.e., the proportion of answers conflicting with facts) through consistency verification of decision-making agents and expert refinement; MMKG embeds visual data storage paths, and web retrieval provides URL tracing, allowing users to trace the source of each fact in the answer, thus solving the black-box reasoning problem in existing RAGs. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a block diagram illustrating the overall principle of the MARAG framework.

[0033] Figure 2 A flowchart of the decision-making intelligent agent's workflow. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Reference Figure 1 As shown, the present invention provides a multimodal agent-enhanced retrieval algorithm, including a multimodal text conversion unit, a vector database construction unit, and a multimodal knowledge graph (MMKG) construction unit.

[0036] Multi-source retrieval agent module: Three sub-components work in parallel, each handling different types of data sources.

[0037] Decision-making agent module: Integrates and refines answers from multiple sources.

[0038] In the above embodiments, the specific analysis method of the multimodal text conversion unit is as follows: Multimodal text conversion: using the BLIP-2 framework, extracting image features through a visual encoder, converting visual information into structured text descriptions by combining a cross-modal alignment module, and introducing a context refinement mechanism.

[0039] In the above embodiments, the vector database construction unit specifically analyzes the following method: merging the original text T with the refined visual structured text description Tv into a multimodal text library, and encoding the multimodal text aggregate using a text encoder. A dimensional vector, stored in a vector database, is used for vector retrieval by intelligent agents.

[0040] It should be noted that the original text is represented as T, the visual structured text description is represented as Tv, and the vector database is such as Milvus.

[0041] In the above embodiments, the specific calculation formula for the multimodal text library is as follows: .

[0042] In the above embodiments, the specific analysis method of the multimodal knowledge graph (MMKG) construction unit is as follows: Multimodal knowledge graph (MMKG) construction: Based on the multimodal text aggregate Tm, entity relation extraction is performed using the LightRAG framework, and the function... Extract Entities relational triples By combining two-level reasoning enhancement, an MMKG is constructed. It also embeds a visual data storage path to support cross-modal tracing.

[0043] It should be noted that the combination of two-level reasoning enhances global retrieval to locate topic clusters and local retrieval to focus on entity associations.

[0044] In the above embodiments, the three sub-components include a vector retrieval agent, a graph retrieval agent, and a network retrieval agent.

[0045] In the above embodiments, the vector retrieval agent's specific analysis method is as follows: Input: Subtask The text form.

[0046] Step 1: Encode using a text encoder query vector .

[0047] Step 2: Calculation With all document vectors in the vector database cosine similarity .

[0048] Step 3: Retrieve documents with the top-k similarity Input a lightweight LLM to generate an answer. Constraining the decoding parameters (top-p=1.0, temperature=0) ensures deterministic output.

[0049] In the above embodiments, the graph retrieval agent's specific analysis method is as follows: Input: Subtasks in text form. .

[0050] Step 1: Large Model Semantic Generation Local keywords and global keywords.

[0051] Step 2: Hybrid Graph-Vector Matching: Align local keywords with MMKG entity attributes, and match global keywords with MMKG relationship patterns.

[0052] Step 3: Subgraph Expansion: Retrieve relevant triples to construct a subgraph This extends to the 1-hop neighbors (Nu, Ne) of the node and edge to ensure structural integrity.

[0053] It should be noted that local keywords are fine-grained entities.

[0054] Furthermore, the specific analysis method of the network retrieval agent is as follows: Input: Subtask The text form.

[0055] Step 1: Call the bochaAPI or other search tools.

[0056] Step 2: Obtain the structured results returned by the API Includes title, abstract, URL, and ranking.

[0057] Step 3: Filter redundant results and extract key information to generate the answer. .

[0058] It should be noted that redundant results, such as duplicate URLs or duplicate content, are filtered out.

[0059] In the above embodiments, the specific analysis method for integrating and refining multi-source answers is as follows: the voting of the large model for filtering user information is verified through experiments, and the threshold for deduplication of information is determined by the proportion and embedding model. Multiple lightweight models vote on the input information to determine whether the retrieved information is relevant to the question. When the proportion of relevant voting results reaches the threshold, the information is retained.

[0060] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications and additions should fall within the protection scope of the present invention.

Claims

1. A multimodal agent-enhanced retrieval algorithm, characterized in that, include: User multimodal query input module: Multimodal knowledge preprocessing. Before the framework runs, multimodal knowledge preprocessing needs to be completed to provide a data foundation for retrieval. This includes a multimodal text conversion unit, a vector database construction unit, and a multimodal knowledge graph (MMKG) construction unit. Multi-source retrieval agent module: Three sub-components work in parallel, each processing different types of data sources; Decision-making agent module: Integrates and refines answers from multiple sources.

2. The multimodal agent-enhanced retrieval algorithm according to claim 1, characterized in that, The specific analysis method for the multimodal text conversion unit is as follows: Multimodal text transformation: The BLIP-2 framework is adopted, which extracts image features through a visual encoder, combines a cross-modal alignment module to transform visual information into structured text descriptions, and introduces a context refinement mechanism.

3. The multimodal agent-enhanced retrieval algorithm described in point 1, characterized in that, The specific analysis method for the vector database construction unit is as follows: The original text T and the refined visual structured text description Tv are merged into a multimodal text library. The multimodal text aggregate is then encoded using a text encoder. A dimensional vector, stored in a vector database, is used for vector retrieval by intelligent agents.

4. The multimodal agent-enhanced retrieval algorithm according to claim 1, characterized in that, The specific calculation formula for the multimodal text library is as follows: .

5. The multimodal agent-enhanced retrieval algorithm according to claim 1, characterized in that, The specific analysis method for the multimodal knowledge graph (MMKG) construction unit is as follows: Construction of a Multimodal Knowledge Graph (MMKG): Based on the multimodal text aggregate Tm, entity relation extraction is performed using the LightRAG framework, and functions are used. Extract Entities relational triples By combining two-level reasoning enhancement, an MMKG is constructed. It also embeds a visual data storage path to support cross-modal tracing.

6. The multimodal agent-enhanced retrieval algorithm according to claim 1, characterized in that, The three sub-components include a vector retrieval agent, a graph retrieval agent, and a network retrieval agent.

7. The multimodal agent-enhanced retrieval algorithm according to claim 1, characterized in that, The specific analysis method for the vector retrieval agent is as follows: Input: Subtask The text form; Step 1: Encode using a text encoder query vector ; Step 2: Calculation With all document vectors in the vector database cosine similarity ; Step 3: Retrieve documents with the top-k similarity Input a lightweight LLM to generate an answer. Constraining the decoding parameters (top-p=1.0, temperature=0) ensures deterministic output.

8. The multimodal agent-enhanced retrieval algorithm according to claim 1, characterized in that, The graph retrieval agent uses the following specific analysis method: Input: Subtasks in text format ; Step 1: Large Model Semantic Generation Local keywords and global keywords; Step 2: Hybrid Graph-Vector Matching: Align local keywords with MMKG entity attributes, and match global keywords with MMKG relationship patterns; Step 3: Subgraph Expansion: Retrieve relevant triples to construct a subgraph This extends to the 1-hop neighbors (Nu, Ne) of the node and edge to ensure structural integrity.

9. The multimodal agent-enhanced retrieval algorithm according to claim 1, characterized in that, The specific analysis method for the network retrieval agent is as follows: Input: Subtask The text form; Step 1: Call the bochaAPI or other search tools; Step 2: Obtain the structured results returned by the API Includes title, abstract, URL, and ranking; Step 3: Filter redundant results and extract key information to generate the answer. .

10. The multimodal agent-enhanced retrieval algorithm according to claim 1, characterized in that, The specific analysis method for integrating and refining multi-source answers is as follows: Experiments were conducted to verify that the large-scale user information filtering model uses a voting mechanism based on proportion and an embedding model to determine the threshold for information deduplication. Multiple lightweight models vote on the input information to determine whether the retrieved information is relevant to the question. When the proportion of relevant votes reaches a threshold, the information is retained.

Images