Multi-modal data retrieval method and device under dynamic collaboration and medium

Abstract

The invention discloses a multi-modal data retrieval method and device under dynamic collaboration and a medium, belongs to the technical field of information retrieval and natural language processing, and is used for solving the problems that unstructured data is difficult to process efficiently in traditional data retrieval, an information island phenomenon exists, and the data retrieval efficiency is low. And an effective connection and integration mechanism is lacked between data from different sources. The method comprises the following steps: performing task allocation processing on a corresponding Agent cluster on a question type to be queried to obtain an Agent task list; performing data query processing on the Agent task list under different data sources, and obtaining query result data; performing data primary processing on the query result data to determine initial retrieval data; performing multi-modal consistency check on the initial retrieval data, and integrating verified integrated retrieval data; and performing information strategy generation processing based on a fact verification mode and an explanation enhancement mode on the integrated retrieval data to obtain final answer information.

Description

Technical Field

[0001] This application relates to the fields of information retrieval and natural language processing, and in particular to a multimodal data retrieval method, device and medium under dynamic collaboration. Background Technology

[0002] With the development of the internet, the amount of global data has exploded, encompassing various types from structured to unstructured, such as text and images. However, traditional search engines and technologies have encountered numerous challenges in processing this complex data, including difficulties in efficiently handling unstructured data and the severe phenomenon of information silos—the lack of effective connections and integration mechanisms between data from different sources. Furthermore, cross-modal information integration remains a pressing problem to be solved.

[0003] Based on the aforementioned issues, there is an urgent need to develop a novel data retrieval augmentation generation system. This system must be able to effectively integrate different types of data and provide accurate answers and services. It also needs to incorporate the latest machine learning, deep learning, and natural language processing technologies to intelligently understand users' query intent and extract the most relevant information from massive datasets. Furthermore, by utilizing dynamic agent collaboration technology combined with multimodal retrieval augmentation generation (RAG) technology, the system can significantly improve its ability to understand complex queries, thereby better meeting users' diverse needs. Summary of the Invention

[0004] This application provides a method, device, and medium for multimodal data retrieval under dynamic collaboration, which addresses the following technical problems: traditional data retrieval is difficult to efficiently process unstructured data, there are information silos, and there is a lack of effective connection and integration mechanisms between data from different sources.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On one hand, this application provides a multimodal data retrieval method under dynamic collaboration, including: based on the user's query request information, performing task allocation processing on the corresponding Agent cluster for the question type to be queried, and obtaining an Agent task list; through the Agent cluster under parallel collaboration, performing data query processing on the Agent task list under different data sources, and obtaining query result data; performing preliminary data processing on the query result data to determine initial retrieval data; performing multimodal consistency verification on the initial retrieval data, judging and integrating verified integrated retrieval data; performing information strategy generation processing on the integrated retrieval data based on fact-checking mode and explanation enhancement mode to obtain final answer information corresponding to the query request information; and dynamically updating the routing strategy Agent and question discrimination Agent through a preset memory and based on the task execution process data of the final answer information to obtain a feedback-optimized Agent collaboration mechanism.

[0007] This application's embodiments construct an efficient, intelligent, and adaptive question-answering and information retrieval platform by introducing mechanisms such as question discrimination, reinforcement learning-driven routing decisions, multi-agent collaborative execution, text-image consistency judgment, and closed-loop feedback optimization. The system can automatically match the optimal retrieval path based on the user's query type and perform collaborative retrieval and fusion analysis across multiple data sources, including text, images, and structured databases, ultimately generating accurate, rich, and interpretable answers. The reinforcement learning-based routing decision module and shared context memory mechanism support dynamic task allocation and continuous strategy optimization. The system can continuously adjust agent collaboration methods and retrieval strategies based on historical task performance and user feedback, achieving continuous performance evolution. Furthermore, the generation module possesses a confidence-driven feedback mechanism, automatically triggering secondary retrieval when key information is insufficient or confidence is low, ensuring answer quality while enhancing the system's intelligence and robustness.

[0008] In one feasible implementation, based on the user's query request information, task allocation processing is performed on the corresponding Agent cluster for the question type to be queried, resulting in an Agent task list. Specifically, this includes: receiving the user's query request information; using a question discrimination Agent to perform information type discrimination processing on the question type in the query request information, obtaining question type information; wherein the question type includes at least: factual, exploratory, and multimodal; using a routing strategy Agent driven by a reinforcement learning model to perform task allocation processing on the question query tasks corresponding to the question type information on the relevant Agent cluster, determining the Agent task list corresponding to each functional Agent; wherein the Agent task list includes: simple factual questions directly pointing to a QA knowledge base Agent; complex multimodal questions using a text retrieval Agent, an image page retrieval Agent, and a consistency judgment Agent.

[0009] In one feasible implementation, a parallel collaborative Agent cluster performs data query processing on the Agent task list from different data sources and obtains query result data. Specifically, this includes: distributing the Agent task list to the corresponding Agent cluster via a routing decision Agent; initiating parallel collaborative execution of data queries within the Agent cluster; performing network search Agent processing on the Agent task list to obtain network query results; performing QA knowledge base Agent processing on the Agent task list to obtain knowledge base query results; performing text retrieval Agent processing on the Agent task list to obtain text query results; performing image page retrieval Agent processing on the Agent task list to obtain image page query results; performing database query Agent processing on the Agent task list to obtain database query results; and integrating the network query results, knowledge base query results, text query results, image page query results, and database query results in parallel to obtain the query result data.

[0010] In one feasible implementation, preliminary data processing is performed on the query result data to determine the initial retrieval data. Specifically, this includes: deduplicating the query result data; performing conflict detection between data sources on the deduplicated query result data and deleting data sources with data conflicts; evaluating the confidence level of the conflict-detected query result data and discarding data content corresponding to confidence levels below a first preset threshold, thereby determining the initial retrieval data.

[0011] In one feasible implementation, the initial retrieval data is subjected to a multimodal consistency check to determine and integrate verified integrated retrieval data. Specifically, this includes: extracting semantic relationships between text and images in the initial retrieval data using a cross-modal attention mechanism; determining the two-dimensional similarity between space and semantics in the initial retrieval data based on the semantic relationships; evaluating the consistency of text and image information in the initial retrieval data according to the two-dimensional similarity to determine credibility and relevance; wherein the consistency evaluation can be performed using a consistency judgment model; initial retrieval data that meets both the credibility and relevance criteria are determined as verified data; and the verified data is integrated and optimized using a preset knowledge fusion module to obtain the integrated retrieval data; wherein the integrated retrieval data includes at least: temporal alignment processing and evidence weighting processing.

[0012] In one feasible implementation, the integrated retrieval data is processed using an information strategy generation process based on a fact-checking mode and an explanation enhancement mode to obtain the final answer information corresponding to the query request information. Specifically, this includes: identifying multi-source evidence data in the integrated retrieval data that meets the criteria of structural integrity, high consistency, and high confidence; inputting the multi-source evidence data into a generation module; prioritizing the output of structured data and verification data from the multi-source evidence data according to the fact-checking mode in the generation module to obtain first-stage strategy information; supplementing the first-stage strategy information with reasoning logic and background knowledge and contextual explanations according to the explanation enhancement mode in the generation module to generate second-stage strategy information; and obtaining the final answer information based on the second-stage strategy information.

[0013] In one feasible implementation, after processing the integrated retrieval data using information strategy generation based on fact-checking mode and explanation enhancement mode to obtain the final answer information corresponding to the query request information, the method further includes: identifying key information corresponding to low credibility in the integrated retrieval data; dynamically feeding back the return request information to the relevant Agent cluster based on the key information, and performing secondary retrieval processing on the question type corresponding to the key information through the Agent cluster to obtain supplementary key information under high credibility; and replacing the key information with the supplementary key information.

[0014] In one feasible implementation, a dynamic update of the routing strategy agent and the question discrimination agent is performed based on the task execution process data of the final answer information using a preset memory, resulting in a feedback-optimized agent collaboration mechanism. Specifically, this includes: synchronously recording and sharing the task execution process data of the final answer information and the feedback process data of the query request information to the memory; wherein the memory supports multi-turn dialogue state maintenance, user preference modeling, and continuous optimization training of reinforcement learning models; dynamically updating the routing strategy agent and the question discrimination agent based on self-evolution and performance improvement using the memory, resulting in a feedback-optimized agent collaboration mechanism; and fine-tuning and updating the consistency judgment model and the generation module using interactive data using the memory, resulting in a feedback-optimized consistency judgment model and generation module.

[0015] Secondly, embodiments of this application also provide a multimodal data retrieval device under dynamic collaboration, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to execute a multimodal data retrieval method under dynamic collaboration as described in any of the above embodiments.

[0016] Thirdly, embodiments of this application also provide a non-volatile computer storage medium, which is a non-volatile computer-readable storage medium storing at least one program. Each program includes instructions, which, when executed by a terminal, cause the terminal to execute a multimodal data retrieval method under dynamic collaboration as described in any of the above embodiments.

[0017] This application provides a method, device, and medium for multimodal data retrieval under dynamic collaboration. Compared with the prior art, the embodiments of this application have the following beneficial technical effects:

[0018] 1. By introducing mechanisms such as question discrimination, reinforcement learning-driven routing decisions, multi-agent collaborative execution, text-image consistency judgment, and closed-loop feedback optimization, a highly efficient, intelligent, and adaptive question-answering and information retrieval platform has been constructed. This system can automatically match the optimal search path based on the user's query type and perform collaborative retrieval and fusion analysis across multiple data sources, including text, images, and structured databases, ultimately generating accurate, rich, and interpretable answers.

[0019] 2. To address the limitations of traditional information retrieval systems in processing unstructured and multimodal data, such as difficulties in cross-modal integration, fragmented information sources, and low retrieval efficiency, this application proposes a multimodal consistency judgment mechanism and a spatial-semantic dual-dimensional similarity calculation model. This effectively improves the consistency verification capability of text and image information and solves the technical bottleneck of the difficulty in uniformly evaluating multi-source heterogeneous data. Simultaneously, a knowledge fusion module is used to comprehensively process retrieval results from multiple sources, including deduplication, conflict detection, and confidence weighting, significantly improving the accuracy and reliability of the output answers.

[0020] 3. To address the lack of dynamic scheduling and self-optimization capabilities in existing systems, this application designs a reinforcement learning-based routing decision module and a shared context memory mechanism, supporting dynamic task allocation and continuous strategy optimization. The system can continuously adjust the agent's collaboration method and retrieval strategy based on historical task performance and user feedback, achieving continuous performance evolution. Furthermore, the generation module possesses a confidence-driven feedback mechanism, automatically triggering a secondary retrieval when key information is insufficient or confidence is low, ensuring answer quality while enhancing the system's intelligence and robustness. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0022] Figure 1 A flowchart illustrating a multimodal data retrieval method under dynamic collaboration provided in this application embodiment;

[0023] Figure 2 A block diagram of a dynamic agent-coordinated multimodal data RAG system provided in this application embodiment;

[0024] Figure 3 This is a schematic diagram of the structure of a multimodal data retrieval device under dynamic collaboration, provided in an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0026] It should be noted that this application relates to a multimodal data retrieval augmented generation (RAG) system based on dynamic agent collaboration, applicable to the fields of information retrieval and natural language processing. This system integrates multimodal data through intelligent agents, optimizes the retrieval process through agent collaboration, and enhances the processing capability for data with different structures by combining the latest RAG technology. It solves the problems of information silos and cross-modal integration, improves retrieval accuracy and efficiency, and expands application scenarios. This system integrates artificial intelligence and machine learning technologies, aiming to provide a more intelligent service experience and enhance data utilization and system flexibility.

[0027] Among them, an agent cluster is a collaborative system composed of multiple intelligent agents, belonging to the categories of distributed artificial intelligence and multi-agent systems (MAS). In the field of computer science, an agent can be regarded as a computer entity that can continuously and autonomously function in a distributed or collaborative system, i.e., a software intelligent agent.

[0028] This application provides a multimodal data retrieval method under dynamic collaboration, such as... Figure 1 As shown, the multimodal data retrieval method under dynamic collaboration specifically includes steps S101-S106:

[0029] S101. Based on the user's query request information, perform task allocation processing on the corresponding Agent cluster for the type of question to be queried, and obtain the Agent task list.

[0030] Specifically, the user's query request information can be received first through the Multimodal Data Retrieval Enhanced Generation (RAG) system.

[0031] Furthermore, the question identification agent performs information type discrimination processing on the question type in the query request information to obtain question type information. The question types include at least: factual, exploratory, and multimodal.

[0032] Furthermore, through a routing strategy agent driven by a reinforcement learning model, the question query tasks corresponding to the question type information are assigned to relevant agent clusters, thus determining the agent task list corresponding to each functional agent. This agent task list includes: simple factual questions directly link to the QA knowledge base agent; complex multimodal questions utilize text retrieval agents, image page retrieval agents, and consistency judgment agents.

[0033] In one embodiment, Figure 2A block diagram of a dynamic agent-coordinated multimodal data RAG system is provided for embodiments of this application, such as... Figure 2 As shown, the question identification module first receives the user's query request and performs preliminary analysis and classification. Based on the question identification agent, it identifies and judges the question type (e.g., factual, exploratory, multimodal, etc.). The routing decision module, driven by a reinforcement learning model, intelligently allocates query tasks to the corresponding agent cluster and generates an agent task list. For example, simple factual questions directly target the QA knowledge base agent; complex multimodal questions simultaneously activate the text retrieval agent, image page retrieval agent, and consistency judgment agent.

[0034] S102. Through the Agent cluster in parallel collaboration, perform data query processing on the Agent task list from different data sources and obtain the query result data.

[0035] Specifically, it is also necessary to use the routing decision agent to distribute the agent task list to the corresponding agent cluster; and to start the parallel collaborative execution of relevant data queries in the agent cluster.

[0036] Furthermore, the Agent task list undergoes processing related to online search to obtain online query results. The Agent task list also undergoes processing related to QA knowledge base to obtain knowledge base query results. The Agent task list undergoes processing related to text retrieval to obtain text query results. The Agent task list undergoes processing related to image page retrieval to obtain image page query results. Finally, the Agent task list undergoes processing related to database query to obtain database query results.

[0037] Furthermore, the online query results, knowledge base query results, text query results, image page query results, and database query results are integrated in parallel to obtain the query result data.

[0038] In one embodiment, such as Figure 2 As shown, during the dynamic agent collaborative execution phase, each functional agent processes user queries in parallel based on the agent task list generated by the routing decision module (routing decision agent). The network search agent, QA knowledge base agent, text retrieval agent, image page retrieval agent, and database query agent obtain information from different data sources to form query result data.

[0039] S103. Perform preliminary data processing on the query results to determine the initial retrieval data.

[0040] Specifically, the query results data must be deduplicated. After deduplication, the query results data undergoes conflict detection between data sources, and data sources with conflicting data are deleted. The query results data after conflict detection is then evaluated for confidence level, and data with confidence levels below a first preset threshold is discarded to determine the initial retrieval data.

[0041] In one embodiment, such as Figure 2 As shown, after each Agent completes the information retrieval, it transmits the results to the content processing module, and performs preliminary processing on the data returned by each Agent, including deduplication, conflict detection, and confidence assessment.

[0042] S104. Perform a multimodal consistency check on the initial retrieval data, and determine and integrate the validated integrated retrieval data.

[0043] Specifically, a cross-modal attention mechanism is first used to extract the semantic relationships between text and images in the initial retrieval data. Based on these semantic relationships, a two-dimensional similarity between spatial and semantic dimensions is determined in the initial retrieval data.

[0044] Furthermore, the consistency of text and image information in the initial search data is evaluated based on two-dimensional similarity to determine credibility and relevance. The consistency assessment can be performed using a consistency judgment model.

[0045] Furthermore, the initial search data that meets both credibility and relevance requirements needs to be identified as verified data. Through a pre-defined knowledge fusion module, the verified data is integrated and optimized to obtain integrated search data. This integrated search data includes at least: time-series alignment processing and evidence weighting processing.

[0046] In one embodiment, such as Figure 2 As shown, the data, after initial processing, enters the multimodal consistency judgment stage. The system utilizes a cross-modal attention mechanism to extract semantic relationships between text and images, and evaluates the consistency of text and image information through a spatial-semantic dual-dimensional similarity calculation model. The score output by the consistency judgment module is used to determine the relevance and credibility of the text and image information in the search results. Subsequently, this verified information is sent to the knowledge fusion module for further integration and optimization, including but not limited to temporal alignment and evidence weighting, to ensure that the final generated content is both comprehensive and accurate.

[0047] S105. Perform information strategy generation processing on the integrated retrieval data based on the fact-checking mode and the explanation enhancement mode to obtain the final answer information corresponding to the query request information.

[0048] Specifically, the process first identifies and integrates multi-source evidence data from the retrieved data that meets the criteria of structural integrity, high consistency, and high confidence. This multi-source evidence data is then input into the generation module.

[0049] Furthermore, based on the fact-checking mode in the generation module, the structured data and verification data in the multi-source evidence data are prioritized for output processing to obtain the first-stage strategy information.

[0050] Furthermore, based on the explanation enhancement mode in the generation module, the first-stage strategy information is supplemented with reasoning logic and background knowledge and contextual explanations to generate the second-stage strategy information.

[0051] Furthermore, based on the information from the second-stage strategy, the final response information is obtained.

[0052] Furthermore, identify and integrate key information corresponding to low credibility in the retrieved data.

[0053] Furthermore, based on the key information, the feedback request information is dynamically fed back to the relevant Agent cluster. The Agent cluster then performs a secondary retrieval process on the question type corresponding to the key information to obtain supplementary key information with higher credibility. Finally, the supplementary key information replaces the original key information.

[0054] In one embodiment, such as Figure 2 As shown, after the knowledge fusion module completes information integration, it inputs structured, consistent, and highly confident multi-source evidence into the generation module. This module adopts a two-stage generation strategy: the first stage is a fact-checking mode, which prioritizes the output of structured data and verified accurate information. The second stage is an explanation enhancement mode, which supplements the reasoning logic, background knowledge, and contextual explanations based on the aforementioned results, generating natural, fluent, and rich final answer information. During the generation process, the system can also dynamically trigger a feedback mechanism based on the output confidence level. If the credibility of key information is insufficient, a request is sent back to the relevant Agent for secondary retrieval to ensure the reliability and completeness of the output results. The entire process is closed-loop and collaborative, achieving end-to-end optimization from problem understanding to intelligent generation.

[0055] S106. Through a preset memory and based on the task execution process data of the final answer information, dynamically update the routing strategy agent and the problem discrimination agent to obtain the optimized agent collaboration mechanism.

[0056] Specifically, the task execution process data of the final answer information and the feedback process data of the query request information are first synchronously recorded and shared in the memory. The memory supports multi-turn dialogue state maintenance, user preference modeling, and continuous optimization training of reinforcement learning models.

[0057] Furthermore, through the memory database, the routing strategy agent and the problem identification agent are dynamically updated for self-evolution and performance improvement, resulting in a feedback-optimized agent collaboration mechanism.

[0058] Furthermore, through the memory, the consistency judgment model and the generation module are fine-tuned and updated with interactive data to obtain the feedback-optimized consistency judgment model and generation module.

[0059] In one embodiment, such as Figure 2 As shown, after the generation module completes the final answer output, the system synchronously records user feedback and data from the entire task execution process to a shared context memory. This memory supports multi-turn dialogue state maintenance, user preference modeling, and continuous optimization training of reinforcement learning models. Through dynamic updates to the routing strategy and agent collaboration mechanism—that is, dynamically updating the routing strategy agent and the question-judgment agent—the system achieves self-evolution and performance improvement, further enhancing its adaptability to complex queries and response accuracy. Simultaneously, user interaction data can also be used for fine-tuning the consistency judgment model and the generation module, thereby continuously improving the system's intelligence level and user experience quality.

[0060] In addition, embodiments of this application also provide a multimodal data retrieval device under dynamic collaboration, such as... Figure 3 As shown, the multimodal data retrieval device 300 under dynamic collaboration specifically includes:

[0061] At least one processor 301. And a memory 302 communicatively connected to the at least one processor 301. The memory 302 stores instructions executable by the at least one processor 301, enabling the at least one processor 301 to execute:

[0062] Based on the user's query request information, the task allocation and processing of the corresponding Agent cluster is carried out according to the type of question to be queried, and the Agent task list is obtained.

[0063] By using a parallel and collaborative Agent cluster, data query processing is performed on the Agent task list from different data sources, and the query results are obtained.

[0064] Perform preliminary data processing on the query results to determine the initial retrieval data;

[0065] The initial retrieval data is subjected to multimodal consistency testing, and the validated integrated retrieval data is determined and integrated.

[0066] The integrated retrieval data is processed using information strategy generation based on fact-checking mode and explanation enhancement mode to obtain the final answer information corresponding to the query request information;

[0067] By using a pre-set memory and based on the task execution process data of the final answer information, the routing strategy agent and the question discrimination agent are dynamically updated to obtain a feedback-optimized agent collaboration mechanism.

[0068] This application constructs an efficient, intelligent, and adaptive question-answering and information retrieval platform by introducing mechanisms such as question discrimination, reinforcement learning-driven routing decision-making, multi-agent collaborative execution, text-image consistency judgment, and closed-loop feedback optimization. The system can automatically match the optimal retrieval path based on the user's query type and perform collaborative retrieval and fusion analysis across multiple data sources, including text, images, and structured databases, ultimately generating accurate, rich, and interpretable answers. The reinforcement learning-based routing decision-making module and shared context memory mechanism support dynamic task allocation and continuous strategy optimization. The system can continuously adjust agent collaboration methods and retrieval strategies based on historical task performance and user feedback, achieving continuous performance evolution. Furthermore, the generation module has a confidence-driven feedback mechanism that automatically triggers secondary retrieval when key information is insufficient or confidence is low, ensuring answer quality while enhancing the system's intelligence and robustness.

[0069] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0070] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0076] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0077] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0078] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of this specification.

Claims

1. A multimodal data retrieval method under dynamic collaboration, characterized in that, The method includes: Based on the user's query request information, the task allocation and processing of the corresponding Agent cluster is carried out according to the type of question to be queried, and the Agent task list is obtained. Through a parallel and collaborative Agent cluster, data query processing is performed on the Agent task list from different data sources, and the query result data is obtained. The query results data are subjected to preliminary data processing to determine the initial retrieval data; The initial retrieval data is subjected to a multimodal consistency test to determine and integrate the verified integrated retrieval data. The integrated retrieval data is processed using information strategy generation based on fact-checking mode and explanation enhancement mode to obtain the final answer information corresponding to the query request information; By using a pre-set memory and based on the task execution process data of the final answer information, the routing strategy agent and the question discrimination agent are dynamically updated to obtain a feedback-optimized agent collaboration mechanism.

2. The multimodal data retrieval method under dynamic collaboration according to claim 1, characterized in that, Based on the user's query request information, tasks are assigned to the corresponding Agent clusters for the type of query to be processed, resulting in an Agent task list, which includes: Receive the user's query request information; The question identification agent performs information type identification processing on the question type in the query request information to obtain question type information; wherein, the question type includes at least: factual, exploratory and multimodal. Using a routing strategy agent driven by a reinforcement learning model, the question query tasks corresponding to the question type information are assigned to relevant agent clusters to determine the agent task list corresponding to each functional agent. The agent task list includes: simple factual questions are directly linked to the QA knowledge base agent; complex multimodal questions use text retrieval agent, image page retrieval agent, and consistency judgment agent.

3. The multimodal data retrieval method under dynamic collaboration according to claim 1, characterized in that, Through a parallel and collaborative agent cluster, data query processing is performed on the agent task list from different data sources, and the query result data is obtained, specifically including: The routing decision agent distributes the agent task list to the corresponding agent cluster; and initiates parallel collaborative execution of relevant data queries in the agent cluster. Perform relevant network search agent processing on the Agent task list to obtain network query results; The Agent task list is processed using QA knowledge base Agent to obtain knowledge base query results; The Agent task list is processed by the relevant text retrieval Agent to obtain text query results; The Agent task list is processed by the relevant image page retrieval Agent to obtain the image page query results; Perform database query processing on the Agent task list to obtain the database query results; The query result data is obtained by integrating the network query results, the knowledge base query results, the text query results, the image page query results, and the database query results in parallel.

4. The multimodal data retrieval method under dynamic collaboration according to claim 1, characterized in that, The query results data undergo preliminary data processing to determine the initial retrieval data, specifically including: The query results data are deduplicated. After deduplication, the query results are checked for conflicts between data sources, and data sources with data conflicts are deleted. The confidence level of the query results after conflict detection is evaluated, and the data content corresponding to the confidence level below the first preset threshold is discarded to determine the initial retrieval data.

5. The multimodal data retrieval method under dynamic collaboration according to claim 1, characterized in that, The initial retrieval data is subjected to a multimodal consistency test to determine and integrate the validated integrated retrieval data, specifically including: The semantic relationships between text and images in the initial retrieval data are extracted using a cross-modal attention mechanism. Based on the semantic association, the two-dimensional similarity between space and semantics in the initial retrieval data is determined; Based on the aforementioned two-dimensional similarity, the consistency of the text and image information in the initial retrieval data is evaluated to determine the credibility and relevance; wherein, the consistency evaluation can be performed using a consistency judgment model. The initial search data that meets both the credibility and relevance criteria are determined as verified data. The verification data is integrated and optimized through a pre-defined knowledge fusion module to obtain the integrated retrieval data; wherein the integrated retrieval data includes at least: time-series alignment processing and evidence weighting processing.

6. The multimodal data retrieval method under dynamic collaboration according to claim 1, characterized in that, The integrated retrieval data is processed using information strategy generation based on fact-checking and explanation enhancement modes to obtain the final answer information corresponding to the query request information, specifically including: Identify multi-source evidence data in the integrated retrieval data that meets the criteria of structural integrity, high consistency, and high confidence; and input the multi-source evidence data into the generation module; Based on the fact verification mode in the generation module, the structured data and verification data in the multi-source evidence data are prioritized for output processing to obtain the first-stage strategy information; Based on the explanation enhancement mode in the generation module, the first-stage strategy information is supplemented with reasoning logic and background knowledge and explanation of contextual information to generate the second-stage strategy information. Based on the strategy information from the second stage, the final answer information is obtained.

7. The multimodal data retrieval method under dynamic collaboration according to claim 6, characterized in that, After processing the integrated retrieval data using information strategy generation based on fact-checking and interpretation enhancement modes to obtain the final answer information corresponding to the query request information, the method further includes: Identify key information corresponding to low confidence levels in the integrated retrieval data; Based on the key information, the feedback request information is dynamically fed back to the relevant Agent cluster, and the Agent cluster is used to perform secondary retrieval processing on the question type corresponding to the key information to obtain supplementary key information with high credibility. Replace the key information with the supplementary key information.

8. The multimodal data retrieval method under dynamic collaboration according to claim 1, characterized in that, By using a pre-set memory and based on the task execution process data of the final answer information, the routing strategy agent and the question discrimination agent are dynamically updated to obtain a feedback-optimized agent collaboration mechanism, specifically including: The task execution process data of the final answer information and the feedback process data of the query request information are synchronously recorded and shared in the memory; wherein, the memory supports multi-turn dialogue state maintenance, user preference modeling, and continuous optimization training of reinforcement learning models; Through the memory, the routing strategy agent and the problem identification agent are dynamically updated under the conditions of self-evolution and performance improvement to obtain the agent cooperation mechanism with feedback optimization. Through the memory, the consistency judgment model and the generation module are fine-tuned and updated with interactive data to obtain the feedback-optimized consistency judgment model and generation module.

9. A multimodal data retrieval device under dynamic collaboration, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform a multimodal data retrieval method under dynamic collaboration as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The storage medium is a non-volatile computer-readable storage medium that stores at least one program, each program including instructions that, when executed by a terminal, cause the terminal to perform a multimodal data retrieval method under dynamic collaboration as described in any one of claims 1-8.

Images