Multi-Agent collaborative question-answering system based on multi-modal large model

Abstract

The invention discloses a multi-Agent collaborative question-answering system based on a multi-modal large model, which comprises an intention understanding Agent and a plurality of task processing Agent libraries, the plurality of task processing Agent libraries comprise an accurate question-answering Agent, a customer complaint Agent, a multi-hop reasoning Agent, a quality inspection Agent and a polling Agent, the intention understanding Agent is configured to receive and analyze user input, and the polling Agent is configured to receive and analyze the user input. Key questions are extracted from user input, the key questions are classified, the real intention of a user is judged, and a corresponding task processing Agent library is triggered according to the key questions; and the plurality of task processing Agent libraries are configured to process complex questions step by step, generate detailed answers layer by layer and perform error correction through a self error correction mechanism, and the plurality of task processing Agent libraries share information through a message queue to realize cooperation.

Description

Technical Field

[0001] This invention relates to the field of intelligent customer service, and more specifically, to a multi-agent collaborative question-and-answer system based on a multimodal large model. Background Technology

[0002] With the rapid development of large-scale models and multi-agent technologies, existing intelligent question-answering systems can handle user requests in various scenarios. However, when faced with complex and diverse user needs, the question-answering capabilities of a single large-scale model or single agent are limited, making it difficult to efficiently handle multimodal (text, audio, image) input and complex problems requiring multi-step reasoning. For example, existing single-agent systems have limited processing capabilities, especially in multi-level and multi-type tasks, often generating erroneous information or missing key content due to excessively long contexts or increased task complexity. Furthermore, traditional question-answering systems often rely on fixed processes, unable to flexibly switch question-answering modes according to user needs, making it difficult to adapt to dynamically changing scenarios.

[0003] Existing agent-based collaborative systems largely rely on unimodal input and pre-defined fixed task flows, such as the method disclosed in CN118132236A. This patent proposes a technical solution for automatic multi-agent task coordination based on a large model. It utilizes a Plan Agent to break down tasks, a Process Agent to extract historical information, and a Review Agent to perform post-mortem analysis. Each agent executes tasks according to a fixed flow, with input and output primarily in text format.

[0004] The patent is implemented in the following ways:

[0005] (1) The task is broken down into multiple steps by the Plan Agent, and each step is handled by a single pre-defined, fixed Agent. Such as rating Agent, car purchase consultant, filing Agent, etc.

[0006] (2) All information in the task execution is stored in the message queue. The Process Agent is responsible for completing the prompt for each step based on the historical messages.

[0007] (3) After the task is completed, the Review Agent will review and debrief the process and output a summary report.

[0008] (4) The system supports multi-step execution of Agent tasks and uses large models such as gpt4, but the input and output are limited to text form and cannot process multimodal information.

[0009] The advantage of this technical solution lies in its versatility. It can handle multiple different tasks through agent collaboration and can dynamically complete the prompt through historical records.

[0010] Although the aforementioned existing technologies have initially achieved multi-agent collaborative work and improved efficiency through automatic task decomposition and review of large models, the following shortcomings still exist:

[0011] (1) Single-modal limitation: Existing systems mainly rely on text input and output and cannot process multimodal information such as images and audio. This is insufficient when processing user queries that contain multimodal inputs (such as image recognition and speech parsing). This invention solves this problem by integrating the processing capabilities of text, images, and audio into a large multimodal model.

[0012] (2) Fixed processes are difficult to handle complex tasks: Existing technologies have fixed processes and cannot flexibly call different agents according to the dynamic changes of the task, making it difficult to handle complex, multi-level problems that require step-by-step thinking. In contrast, this invention can flexibly switch the processing flow according to the difficulty of the task and the level of the problem through a dynamic multi-hop reasoning chain and agent collaboration mechanism.

[0013] (3) The quality inspection mechanism is only for later evaluation: Although there is a quality inspection mechanism (Review Agent) in this invention, the output of the quality inspection agent is only used to evaluate whether there are problems with the session records in the later stage, and cannot control the accuracy of the output results of the large model in real time. There is a lag between quality inspection and correction.

[0014] In existing technologies, such as CN118396119A, a multimodal large-scale model-based intelligent agent task data processing system is proposed. The core of this technology is to process task data through a multimodal large-scale model-based intelligent agent system, primarily applied to question-and-answer systems in the enterprise customer service field. Its specific implementation scheme includes:

[0015] (1) Multimodal processing: The system preprocesses multimodal data such as text, voice or images input by the customer to convert them into a unified preprocessed data format.

[0016] (2) Task tag generation: The preprocessed data undergoes semantic processing through the enterprise big data model to generate corresponding task tags. Task tags include information such as problem category, customer sentiment, and named entities.

[0017] (3) Keyword extraction and similarity calculation: The agent extracts keywords from the generated response information and calculates the matching degree between the task tag and the keywords. If the matching degree exceeds the preset threshold, the system directly generates and sends the response information; otherwise, it is transferred to human customer service for processing.

[0018] (4) Data cleaning and incremental training: The agent performs semantic processing on the processed customer questions and human customer service responses, and stores the results in the memory bank. Data cleaning is performed regularly, and the cleaned high-quality data is used as incremental training data to optimize the enterprise large model.

[0019] The key features of this technology are its automatic data cleaning and incremental knowledge base training capabilities, which greatly improve the optimization speed and response accuracy of enterprise large models.

[0020] Although this technology performs well in enterprise customer service scenarios by processing and optimizing data through multimodal large models, it still has the following shortcomings:

[0021] (1) Single-task processing: This system is mainly designed for specific tasks of enterprise customer service and is difficult to deal with complex multi-task and multi-domain problems. In contrast, this invention can dynamically adapt to different types of tasks through multi-agent collaboration, thus expanding the application scenarios.

[0022] (2) Lack of multi-hop reasoning ability: Existing question-answering systems rely on keyword matching and semantic analysis, but lack a multi-hop reasoning chain mechanism, making it difficult to solve complex problems that require multiple derivations. This invention solves this limitation by using a multi-hop reasoning agent to gradually break down complex problems into multiple sub-problems and process them recursively.

[0023] (3) Lack of quality control mechanism: Existing systems lack a quality control mechanism after generating responses, which may lead to some answers being inaccurate. This invention ensures the quality of the output answers through a quality control agent, thereby improving user satisfaction and system reliability. Summary of the Invention

[0024] This invention provides a multi-agent collaborative question-answering system based on a multimodal large model to solve the technical problems existing in the prior art.

[0025] To achieve the above objectives, this invention provides a multi-agent collaborative question-answering system based on a multimodal large model, comprising: an intent understanding agent and multiple task processing agent libraries, the multiple task processing agent libraries including an accurate question-answering agent, a customer complaint agent, a multi-hop reasoning agent, a quality inspection agent, and an inquiry agent.

[0026] The intent understanding agent is configured to receive and parse user input, extract key questions from the user input, classify the key questions and determine the user's true intent, and trigger the corresponding task processing agent library accordingly.

[0027] Multiple task processing agent libraries are configured to process complex problems step by step, generate detailed answers layer by layer, and correct errors through a self-correction mechanism. The multiple task processing agent libraries share information through message queues to achieve collaboration.

[0028] In one embodiment of the present invention, the user input integrates multimodal information, which includes at least one of text, audio, and image. Parsing the user input includes analyzing natural language, parsing audio, and processing images.

[0029] In one embodiment of the present invention, the user's true intention includes inquiry, complaint or casual conversation, wherein inquiry includes product / service inquiry and contact information inquiry.

[0030] In one embodiment of the present invention, the precise question-answering agent internally includes a retrieval module, a knowledge base, and a relevance judgment module. Multimodal information processed by the intent understanding agent and the knowledge base are input into the retrieval module. The retrieval module outputs retrieval results, which are preset knowledge content in the knowledge base related to the user's input. Subsequently, the relevance judgment module outputs an assertion on whether the retrieval results are relevant to the user's question. If the retrieval results are irrelevant to the user's input, the multimodal information and the retrieval results are further input into the inquiry agent. If the retrieval results are relevant to the user's input, the retrieval results, the question-answering prompt, and the multimodal information are input into the multi-hop reasoning agent. The result processed and answered by the multi-hop reasoning agent is further sent to the quality inspection agent for quality inspection and correction. The result after quality inspection and correction is output as the final answer to the user.

[0031] The multi-hop reasoning agent consists of an Lv1 multimodal big model module and an Lv2 multimodal big model module. The Lv1 multimodal big model module is responsible for decomposing the question and reviewing the answer output by the Lv2 multimodal big model module. First, the multimodal information processed by the intent recognition agent, the retrieval results obtained from the knowledge base, and the question splitting prompt are sent to the Lv1 multimodal big model module. The Lv1 multimodal big model module divides the user's question into several information blocks, and these information blocks are sent to the Lv2 multimodal big model module for sub-question answering. After receiving the answer from the Lv2 multimodal big model module, the Lv1 multimodal big model module considers whether other information is needed or whether the current answer is accurate based on the original question and the answer of the previous step. This process is repeated until the question is completely answered, and finally the answer is output to the next-level task processing agent library. The above loop process automatically forms a thought chain for complex nested questions and decomposes them into multi-step reasoning, generating detailed answers layer by layer.

[0032] The quality inspection agent receives the answer returned by the multi-hop processing agent, scores the answer based on the knowledge base, determines whether the answer can effectively solve the user's problem, and revises the answer accordingly.

[0033] The Exploration Agent receives questions that cannot be found in the knowledge base, questions that users want to chat about, or questions that cannot be categorized. If a user's question cannot be answered in the knowledge base, the Exploration Agent guides the user to provide more information about the question. If the user just wants to chat, the Exploration Agent guides the user to ask questions that can be solved and refuses to answer related chat topics.

[0034] The Customer Complaint Agent receives customer complaints, categorizes them, and assigns them to different types of human customer service representatives for processing. After processing, the Customer Complaint Agent automatically monitors the outcome. If the processing is successful and the user is satisfied, the relevant historical records are sent to a multimodal database for integration. The integrated results are stored in a knowledge base. When the same problem arises again, the multimodal database can retrieve the relevant information and provide an automatic answer. If the user is not satisfied after processing, the customer's reserved information is sent to a specialist for further handling.

[0035] In one embodiment of the present invention, a multi-agent collaborative question-answering system with a multimodal large model runs multiple multimodal large models in parallel, including OpenAI-o1, GPT-4o, Qwen2-VL+Qwen2-Audio, and LLaVA-NeXT, to process multimodal information.

[0036] The multi-agent collaborative question-answering system based on a multimodal large model provided by this invention has the following beneficial technical effects:

[0037] 1. Improved multimodal information processing capabilities

[0038] Most existing question-answering systems can only process single-modal data (such as text) and cannot handle inputs containing multiple forms such as images and audio. This invention, by integrating a multimodal large model, achieves effective parsing and processing of various information forms such as text, images, and audio, greatly improving the system's ability to handle complex, multimodal tasks and enabling the system to be more widely applied in real-world scenarios, such as multimedia queries and intelligent customer service.

[0039] 2. Efficient processing mechanism for multi-agent collaboration

[0040] This invention introduces multiple dedicated agents to dynamically allocate tasks and collaboratively fulfill users' multimodal question-answering needs. Different agents (such as intent understanding agents, task processing agents, multi-hop inference agents, and quality inspection agents) share information through message queues and are flexibly invoked according to the complexity of the task. This collaborative mechanism enables the system to efficiently handle multi-step problems, enhancing the flexibility and efficiency of task processing.

[0041] 3. Multi-hop reasoning chain mechanism enhances problem-solving ability.

[0042] Existing question-answering systems often struggle with complex questions requiring multi-step reasoning. This invention utilizes a multi-hop reasoning agent to break down complex problems into several sub-problems and solve them step by step through recursive reasoning. This mechanism effectively improves the system's depth and accuracy in handling complex problems, particularly excelling in cross-domain complex issues.

[0043] 4. Quality inspection and feedback mechanisms ensure output quality.

[0044] This invention utilizes a quality control agent to perform quality checks on each answer, ensuring that the output answers meet user requirements and are highly accurate. The quality control agent not only filters out errors but also adjusts subsequent task processing strategies based on feedback, significantly improving user satisfaction and system reliability.

[0045] In summary, the technical solution of this invention solves the problems of insufficient ability to handle complex tasks, poor dynamic adaptability, and low answer accuracy of existing question-answering systems by using innovative technologies such as multimodal information processing, multi-agent collaboration, multi-hop reasoning chain, and quality inspection feedback, thereby greatly improving the processing efficiency and user experience of intelligent question-answering systems. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is an overall flowchart of a multi-agent collaborative question-answering system based on a multimodal large model according to an embodiment of the present invention;

[0048] Figure 2 This is a flowchart illustrating the intent understanding of an Agent according to an embodiment of the present invention;

[0049] Figure 3This is a flowchart of a precise question-answering agent according to an embodiment of the present invention;

[0050] Figure 4 This is a flowchart of a multi-hop inference agent according to an embodiment of the present invention;

[0051] Figure 5 This is a flowchart of a quality inspection agent according to an embodiment of the present invention;

[0052] Figure 6 This is a flowchart of an embodiment of the present invention, namely, the Query Agent.

[0053] Figure 7 This is a flowchart of a customer complaint agent according to an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Figure 1 This is an overall flowchart of a multi-agent collaborative question-answering system based on a multimodal large model according to an embodiment of the present invention, as follows: Figure 1 As shown, this invention provides a multi-agent collaborative question-answering system based on a multimodal large model, comprising: an intent understanding agent and multiple task processing agent libraries, the multiple task processing agent libraries including a precise question-answering agent, a customer complaint agent, a multi-hop reasoning agent, a quality inspection agent, and an inquiry agent.

[0056] Figure 2 This is a flowchart of an embodiment of the present invention for understanding an Agent, such as... Figure 2 As shown, the Intent Understanding Agent is configured to receive and parse user input, extract key questions from the user input, classify the key questions and determine the user's true intent, and trigger the corresponding task processing Agent library accordingly.

[0057] Multiple task processing agent libraries are configured to process complex problems step by step, generate detailed answers layer by layer, and correct errors through a self-correction mechanism. The multiple task processing agent libraries share information through message queues to achieve collaboration.

[0058] In one embodiment of the present invention, the user input integrates multimodal information, which includes at least one of text, audio, and image. Parsing the user input includes analyzing natural language, parsing audio, and processing images.

[0059] In one embodiment of the present invention, the user's true intention includes inquiry, complaint or casual conversation, wherein inquiry includes product / service inquiry and contact information inquiry.

[0060] Figure 3 This is a flowchart of a precise question-answering agent according to an embodiment of the present invention, such as... Figure 3 As shown, the precise question-answering agent has an internal retrieval module, a knowledge base, and a relevance judgment module. The multimodal information processed by the intent understanding agent and the knowledge base are input into the retrieval module. The retrieval module outputs retrieval results, which are preset knowledge content about the user's input from the knowledge base. Then, the relevance judgment module outputs an assertion on whether the retrieval results are relevant to the user's question. If the retrieval results are not relevant to the user's input, the multimodal information and retrieval results are further input into the inquiry agent. If the retrieval results are relevant to the user's input, the retrieval results, the question-answering prompt, and the multimodal information are input into the multi-hop reasoning agent. The result processed and answered by the multi-hop reasoning agent is further sent to the quality control agent for quality control and correction. The result after quality control and correction is output as the final answer to the user.

[0061] Figure 4 This is a flowchart of a multi-hop inference agent according to an embodiment of the present invention, such as... Figure 4 As shown, the multi-hop reasoning agent consists of an Lv1 multimodal large model module and an Lv2 multimodal large model module. The Lv1 multimodal large model module is responsible for decomposing the question and reviewing the answer output by the Lv2 multimodal large model module. First, the multimodal information processed by the intent recognition agent, the retrieval results obtained from the knowledge base, and the question splitting prompt are sent to the Lv1 multimodal large model module. The Lv1 multimodal large model module divides the user's question into several information blocks, and these information blocks are sent to the Lv2 multimodal large model module for sub-question answering. After receiving the answer from the Lv2 multimodal large model module, the Lv1 multimodal large model module considers whether other information is needed or whether the current answer is accurate based on the original question and the answer from the previous step. This process is repeated until the question is completely answered, and finally the answer is output to the next-level task processing agent library. The above cyclical process automatically forms a thought chain for complex nested questions and decomposes them into multi-step reasoning, generating detailed answers layer by layer.

[0062] Figure 5 Here is a flowchart of a quality inspection agent according to an embodiment of the present invention, as follows: Figure 5As shown, the quality inspection agent, after receiving the answer returned by the multi-hop processing agent, scores the answer based on the knowledge base to determine whether the answer can effectively solve the user's problem and revise the answer accordingly.

[0063] Figure 6 This is a flowchart of an embodiment of the query agent of the present invention, such as... Figure 6 As shown, the Exploration Agent receives questions that are not found in the knowledge base, questions that users want to chat about, or questions that cannot be categorized. If a user's question cannot be answered in the knowledge base, the Exploration Agent guides the user to provide more information about the question. If the user just wants to chat, the Exploration Agent guides the user to ask questions that can be solved and refuses to answer related chat topics.

[0064] Figure 7 This is a flowchart of a customer complaint agent according to an embodiment of the present invention, such as... Figure 7 As shown, the Customer Complaint Agent receives customer complaints, categorizes the user's requests, and assigns them to different types of human customer service representatives for processing. After processing, the Customer Complaint Agent automatically monitors the problem handling results. If the handling is completed and the user is satisfied, the relevant historical records will be sent to the multimodal big data model for historical record integration. The integrated results will be stored in the knowledge base. When the same problem is encountered again, the multimodal big data model can retrieve the relevant content and automatically answer it. If the user is not satisfied after the handling is completed, the customer's reserved information will be pushed to a specialist for handling.

[0065] In this invention, the "prompt" in "question and answer prompt", "exploration prompt", and "single-step answer prompt" can be translated as "prompt word" in Chinese. The difference is that they are different prompts or requirements for the execution instructions of large models. Those skilled in the art know their meaning, so they will not be elaborated here.

[0066] Example 1:

[0067] Q&A Prompt: You are a *** assistant, skilled at solving *** problems. Based on the user's question, ***, the following information was retrieved from the existing knowledge base: ***. Please answer the user's question according to the following requirements. Specific requirements are as follows: 1. ***; 2. ***…

[0068] Example 2:

[0069] Prompt: You are a *** assistant, skilled at solving *** problems. The user's question is ***, and no information can be found in the existing knowledge base. Please guide the user to ask questions that this system can solve, and refuse to answer irrelevant casual questions. Specific requirements are as follows: 1. ***; 2. ***…

[0070] Special Notes: 1. The examples above are for illustrative purposes only and should not be considered as actual implementation details of the invention. 2. The "***" in the examples above represents omitted content, which should be replaced according to the actual situation.

[0071] In one embodiment of the present invention, a multi-agent collaborative question-answering system with a multimodal large model runs multiple multimodal large models in parallel, including OpenAI-o1, GPT-4o, Qwen2-VL+Qwen2-Audio, and LLaVA-NeXT, to process multimodal information.

[0072] The multi-agent collaborative question-answering system based on a multimodal large model provided by this invention has the following beneficial technical effects:

[0073] 1. Improved multimodal information processing capabilities

[0074] Most existing question-answering systems can only process single-modal data (such as text) and cannot handle inputs containing multiple forms such as images and audio. This invention, by integrating a multimodal large model, achieves effective parsing and processing of various information forms such as text, images, and audio, greatly improving the system's ability to handle complex, multimodal tasks and enabling the system to be more widely applied in real-world scenarios, such as multimedia queries and intelligent customer service.

[0075] 2. Efficient processing mechanism for multi-agent collaboration

[0076] This invention introduces multiple dedicated agents to dynamically allocate tasks and collaboratively fulfill users' multimodal question-answering needs. Different agents (such as intent understanding agents, task processing agents, multi-hop inference agents, and quality inspection agents) share information through message queues and are flexibly invoked according to the complexity of the task. This collaborative mechanism enables the system to efficiently handle multi-step problems, enhancing the flexibility and efficiency of task processing.

[0077] 3. Multi-hop reasoning chain mechanism enhances problem-solving ability.

[0078] Existing question-answering systems often struggle with complex questions requiring multi-step reasoning. This invention utilizes a multi-hop reasoning agent to break down complex problems into several sub-problems and solve them step by step through recursive reasoning. This mechanism effectively improves the system's depth and accuracy in handling complex problems, particularly excelling in cross-domain complex issues.

[0079] 4. Quality inspection and feedback mechanisms ensure output quality.

[0080] This invention utilizes a quality control agent to perform quality checks on each answer, ensuring that the output answers meet user requirements and are highly accurate. The quality control agent not only filters out errors but also adjusts subsequent task processing strategies based on feedback, significantly improving user satisfaction and system reliability.

[0081] In summary, the technical solution of this invention solves the problems of insufficient ability to handle complex tasks, poor dynamic adaptability, and low answer accuracy of existing question-answering systems by using innovative technologies such as multimodal information processing, multi-agent collaboration, multi-hop reasoning chain, and quality inspection feedback, thereby greatly improving the processing efficiency and user experience of intelligent question-answering systems.

[0082] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0083] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent collaborative question-answering system based on a multimodal large model, characterized in that, include: The intent is to understand the agent and a library of multiple task processing agents, including a precise question answering agent, a customer complaint agent, a multi-hop inference agent, a quality inspection agent, and an inquiry agent. The intent understanding agent is configured to receive and parse user input, extract key questions from the user input, classify the key questions and determine the user's true intent, and trigger the corresponding task processing agent library accordingly. Multiple task processing agent libraries are configured to process complex problems step by step, generate detailed answers layer by layer, and correct errors through a self-correction mechanism. The multiple task processing agent libraries share information through message queues to achieve collaboration.

2. The multi-agent collaborative question-answering system based on a multimodal large model according to claim 1, characterized in that, User input integrates multimodal information, which includes at least one of text, audio, and images. Parsing user input includes analyzing natural language, parsing audio, and processing images.

3. The multi-agent collaborative question-answering system based on a multimodal large model according to claim 1, characterized in that, A user's true intentions may include inquiries, complaints, or casual conversation. Inquiries include product / service inquiries and inquiries about contact information.

4. The multi-agent collaborative question-answering system based on a multimodal large model according to claim 2, characterized in that, The precise question-answering agent has an internal retrieval module, a knowledge base, and a relevance judgment module. Multimodal information processed by the intent understanding agent and the knowledge base are input into the retrieval module. The retrieval module outputs retrieval results, which are preset knowledge content in the knowledge base related to the user's input. Then, the relevance judgment module outputs an assertion on whether the retrieval results are relevant to the user's question. If the retrieval results are not relevant to the user's input, the multimodal information and retrieval results are further input into the inquiry agent. If the retrieval results are relevant to the user's input, the retrieval results, the question-answering prompt, and the multimodal information are input into the multi-hop reasoning agent. The result processed and answered by the multi-hop reasoning agent is further sent to the quality control agent for quality control and correction. The result after quality control and correction is output as the final answer to the user. The multi-hop reasoning agent consists of an Lv1 multimodal big model module and an Lv2 multimodal big model module. The Lv1 multimodal big model module is responsible for decomposing the question and reviewing the answer output by the Lv2 multimodal big model module. First, the multimodal information processed by the intent recognition agent, the retrieval results obtained from the knowledge base, and the question splitting prompt are sent to the Lv1 multimodal big model module. The Lv1 multimodal big model module divides the user's question into several information blocks, and these information blocks are sent to the Lv2 multimodal big model module for sub-question answering. After receiving the answer from the Lv2 multimodal big model module, the Lv1 multimodal big model module considers whether other information is needed or whether the current answer is accurate based on the original question and the answer of the previous step. This process is repeated until the question is completely answered, and finally the answer is output to the next-level task processing agent library. The above loop process automatically forms a thought chain for complex nested questions and decomposes them into multi-step reasoning, generating detailed answers layer by layer. The quality inspection agent receives the answer returned by the multi-hop processing agent, scores the answer based on the knowledge base, determines whether the answer can effectively solve the user's problem, and revises the answer accordingly. The Exploration Agent receives questions that cannot be found in the knowledge base, questions that users want to chat about, or questions that cannot be categorized. If a user's question cannot be answered in the knowledge base, the Exploration Agent guides the user to provide more information about the question. If the user just wants to chat, the Exploration Agent guides the user to ask questions that can be solved and refuses to answer related chat topics. The Customer Complaint Agent receives customer complaints, categorizes them, and assigns them to different types of human customer service representatives for processing. After processing, the Customer Complaint Agent automatically monitors the outcome. If the processing is successful and the user is satisfied, the relevant historical records are sent to a multimodal database for integration. The integrated results are stored in a knowledge base. When the same problem arises again, the multimodal database can retrieve the relevant information and provide an automatic answer. If the user is not satisfied after processing, the customer's reserved information is sent to a specialist for further handling.

5. The multi-agent collaborative question-answering system based on a multimodal large model according to claim 4, characterized in that, The multi-agent collaborative question-answering system for multimodal large models runs multiple multimodal large models in parallel, including OpenAI-o1, GPT-4o, Qwen2-VL+Qwen2-Audio, and LLaVA-NeXT, to process multimodal information.

Images