Multi-modal interaction method, device and equipment
By constructing a central scheduling module to uniformly coordinate the multimodal interaction architecture of multiple independent functional agents, the problems of low efficiency and poor reliability of multimodal interaction in existing technologies are solved, and efficient and accurate multimodal interaction and system scalability are improved.
Patent Information
- Application Number
- CN202510836552.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-21
- Publication Date
- 2025-11-21
AI Technical Summary
In existing technologies, multimodal interaction schemes based on end-to-end unified multimodal models and large language models (LLM) combined with external tools are difficult to generate multiple modal contents efficiently and reliably in a single model, and have problems such as high computational resource consumption, high training difficulty, and long iteration cycle.
A multimodal interaction architecture is constructed, which is uniformly coordinated by a central scheduling module and collaborates with multiple independent functional agents. By decomposing user requests into multiple sub-tasks and assigning them to corresponding intelligent agents for processing, the system achieves high modularity and low coupling, thereby improving the system's scalability and development iteration efficiency.
It achieves efficient and accurate multimodal interaction, improves the system's scalability, maintainability and development iteration efficiency, can dynamically manage complex serial and parallel task flows, and enhances the system's robustness and intelligence when facing complex and ever-changing user needs and abnormal situations.
Smart Images
Figure CN120994880A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-modal interaction method, device and equipment. BACKGROUND
[0002] With the rapid development of artificial intelligence technology, users' demand for multi-modal interaction with AI systems is increasing. Users expect to be able to input information through various ways such as text, voice, image, video, etc., and obtain comprehensive feedback containing various information modalities (such as text, image, audio, etc.).
[0003] In related technologies, multi-modal interaction is implemented based on an end-to-end unified multi-modal model. Such a model aims to process inputs of multiple modalities and generate outputs of multiple modalities within a single, integrated architecture, and usually uses an encoder to map information of different modalities to a unified semantic space for understanding. However, such a model faces challenges in practical applications. In terms of multi-modal generation, it is extremely difficult to generate various contents such as text, image, voice, etc. in a single model in a balanced and high-quality manner, and it is often difficult to simultaneously ensure the generation effect and quality of various modalities, which cannot meet the demand for efficient and reliable execution. SUMMARY
[0004] The present application provides a multi-modal interaction method, device and equipment, which constructs a multi-modal interaction architecture coordinated by a central scheduling module and cooperated by multiple independent functional agents, realizes high modularization of the system and low coupling degree between components, and improves the scalability, maintainability and development iteration efficiency of the system. In the process of multi-modal interaction with the user, the central scheduling model, based on the understanding of the user's request information, disassembles the user's request information into multiple sub-tasks, and distributes the multiple sub-tasks to corresponding agents for processing, effectively realizing the processing of sub-tasks of different modalities, and achieving the effect of efficient and accurate multi-modal interaction with the user.
[0005] In a first aspect, the present application provides a multi-modal interaction method applied to a central scheduling module in a multi-modal interaction system, comprising the following steps.
[0006] Obtaining request information of a user; According to the request information, sending sub-tasks of corresponding modalities to each agent in a plurality of agents; each agent in the plurality of agents is used to process sub-tasks of different modalities; Receiving execution results of sub-tasks of different modalities sent by the plurality of agents.
[0007] According to the multi-modal interaction method provided by the present application, the step of sending sub-tasks of corresponding modalities to each agent in a plurality of agents according to the request information comprises: According to the request information, a plurality of sub-tasks of a plurality of modalities corresponding to the request information are generated, and an execution plan of the plurality of sub-tasks of the plurality of modalities is determined; According to the execution plan of the plurality of sub-tasks of the plurality of modalities, a sub-task of a corresponding modality is sent to each agent in the plurality of agents.
[0008] According to the multi-modal interaction method provided by the application, the execution plan of the plurality of sub-tasks of the plurality of modalities includes at least one of the following: The execution mode of the plurality of sub-tasks of the plurality of modalities, the dependency relationship and execution order between the plurality of sub-tasks of the plurality of modalities, and the data flow information between the plurality of sub-tasks of the plurality of modalities.
[0009] According to the multi-modal interaction method provided by the application, the method further includes: Obtaining target identifiers of various resources required for execution of each sub-task from a session context and resource management module; Sending the target identifiers to each agent.
[0010] In a second aspect, the application provides a multi-modal interaction method applied to any agent in a multi-modal interaction system, including the following steps.
[0011] Receiving a sub-task of a corresponding modality of the agent sent by a central scheduling module; Executing the sub-task of the corresponding modality, and sending an execution result of the sub-task of the corresponding modality to the central scheduling module.
[0012] According to the multi-modal interaction method provided by the application, the method further includes: Receiving target identifiers of various resources required for execution of the sub-task sent by the central scheduling module; According to the target identifiers, obtaining various resources required for execution of the sub-task from a session context and resource management module, and executing the sub-task of the corresponding modality of the agent sent by the central scheduling module.
[0013] The application also provides a multi-modal interaction device, including the following modules: An obtaining module is configured to obtain request information of a user; A processing module is configured to send a sub-task of a corresponding modality to each agent in a plurality of agents according to the request information, and each agent in the plurality of agents is configured to process a sub-task of a different modality; A receiving module is configured to receive an execution result of a sub-task of a different modality sent by the plurality of agents.
[0014] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-modal interaction method according to any one of the above when executing the program.
[0015] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the multi-modal interaction method according to any one of the above.
[0016] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the multi-modal interaction method according to any one of the above.
[0017] The multi-modal interaction method, device and electronic device provided by the application construct a multi-modal interaction architecture coordinated by a central scheduling module and cooperated by multiple independent function agents, realize high modularization of the system and low coupling degree between components, and improve the scalability, maintainability and development iteration efficiency of the system. In the multi-modal interaction process with the user, the central scheduling model understands the user request information, disassembles the user request information into multiple sub-tasks, and allocates the multiple sub-tasks to corresponding agents for processing, effectively realizes processing of sub-tasks of different modes, and achieves the effect of efficient and accurate multi-modal interaction with the user. BRIEF DESCRIPTION OF DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.
[0019] Figure 1 is one of the flowcharts of the multi-modal interaction method provided by the application.
[0020] Figure 2 is a structural schematic diagram of the multi-modal interaction system provided by the application.
[0021] Figure 3 is the second flowchart of the multi-modal interaction method provided by the application.
[0022] Figure 4 is a structural schematic diagram of the multi-modal interaction device provided by the application.
[0023] Figure 5 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION
[0024] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.
[0025] The technical solutions of the present application will be described below in conjunction with the drawings in the present application. Figures 1-5 The multi-modal interaction method, device and equipment of the present application are described.
[0026] In order to facilitate a clearer understanding of the technical solutions of the embodiments of the present application, some technical contents related to the embodiments of the present application will be introduced first.
[0027] With the rapid development of artificial intelligence technology, users' demand for multi-modal interaction with AI systems is growing. Users expect to be able to input information through various ways such as text, speech, image, video, and obtain comprehensive feedback containing multiple information modalities (such as text, image, audio, etc.).
[0028] Currently, the technical path to achieve such multi-modal interaction mainly includes: Path one: End-to-end unified multi-modal model: Such models aim to process multiple modal inputs and generate multiple modal outputs within a single, integrated architecture. They usually use encoders (e.g. Vision Transformer for visual information, Whisper for speech information) to map information of different modalities to a unified semantic space for understanding. However, such models face challenges in practical applications: first, in multi-modal generation, it is extremely difficult to generate text, images, speech and other content in a single model with balanced and high quality, and it is often difficult to ensure the generation effect and quality of each type of modality at the same time; second, building and training such large unified models requires massive multi-modal data sets, and the consumption of computing resources is huge, and the cycle of model iteration and optimization is longer.
[0029] Path two: taking large language model (LLM) as the core, combined with external tools (Function Call): This scheme takes large language model (LLM) as the core semantic understanding and text generation unit. When the system needs to process or generate information of other modalities besides text (for example, text-to-image generation, text-to-speech conversion), LLM calls external specialized models or tools through the "FunctionCall" mechanism. LLM is responsible for understanding user intent, deciding when to call which tool, and integrating the results returned by the tool before presenting them to the user. Although this path achieves multi-modal capabilities to some extent, there are still shortcomings: first, the traditional Function Call mechanism has limited planning and scheduling capabilities when dealing with tasks that require multiple tools to work together (especially parallel execution or complex dependencies), and often only supports simple serial calls; second, each time a new or modified external tool function is added, if the accuracy of some tool Agent calls is low, the LLM Prompt may need to be retrained or adjusted. The former requires a lot of training resources and time, and careful balancing of different task data ratios is needed to avoid capability degradation, which is difficult to train; the latter is difficult to guarantee good results. In particular, existing LLM-based tool calling schemes, when faced with the need to simultaneously execute multiple independent sub-tasks (parallel calls), or the need to strictly follow a specific order to execute dependent sub-tasks (serial calls), and there may be complex data flow and conditional branches between these sub-tasks, the planning and control capabilities of LLM itself often cannot meet the needs of efficient and reliable execution.
[0030] Figure 1 is one of the flowcharts of the multi-modal interaction method provided by the present application, as shown in Figure 1 The method comprises the following steps: Step 101, obtaining request information of a user.
[0031] Specifically, in the embodiment of the present application, the request information of the user is first obtained. Optionally, the request information contains at least one of a plurality of information modalities. For example, the request information includes text A and image B, and the text A is used to indicate that the user needs to query the weather condition in the image B.
[0032] Step 102, according to the request information, sending a corresponding modal sub-task to each agent in a plurality of agents; each agent in the plurality of agents is used to process a sub-task of a different modality.
[0033] Specifically, after obtaining the request information of the user, the embodiment of the application sends the sub-tasks of the corresponding modal to each agent in the plurality of agents according to the request information of the user. That is, in the embodiment of the application, the user request information is disassembled into multiple sub-tasks based on the understanding of the user request information, and the multiple sub-tasks are allocated to the corresponding agents for processing. Optionally, different agents are used to process sub-tasks of different modalities, thereby effectively achieving efficient and accurate processing of sub-tasks of different modalities. Optionally, the trained central scheduling module can accurately disassemble multiple sub-tasks of multiple modalities based on the understanding of the user request information through reinforcement learning.
[0034] For example, the obtained user request information includes text A and image B, and the text A is used to indicate that the user needs to query the real-time weather condition of the geographic location of the image B. Based on the understanding of the user request information, the user request information is disassembled into sub-task 1 and sub-task 2, sub-task 1 is to recognize the image B to obtain the location information of the image B. Sub-task 2 is to query the weather condition according to the location information of the image B and output the query result. Optionally, agent X is used for image processing, and agent Y is used for text information query and text output. Then, sub-task 1 can be allocated to agent X for processing, and sub-task 2 can be allocated to agent Y for processing, thereby effectively achieving efficient and accurate processing of sub-tasks of different modalities.
[0035] Step 103, receiving the execution results of the sub-tasks of different modalities sent by the plurality of agents.
[0036] Specifically, after completing the corresponding sub-tasks of each modal, the plurality of agents can send the sub-task execution results of each modal to the central scheduling module. The central scheduling module combines the sub-task execution results generated after the execution of all related agents Agent, after necessary integration and formatting, into a unified, multi-modal response, and presents it to the user through the user interaction interface, thereby efficiently and accurately realizing multi-modal interaction with the user.
[0037] It should be noted that the innovative construction of the central scheduling--multi-agent Agent collaboration architecture in the embodiment of the application realizes the high modularization and low coupling of the system, encapsulates various modal processing capabilities including LLM as independent, pluggable functional agents, and uniformly, flexibly and dynamically plans and schedules by a dedicated central scheduling model. This architecture enables each functional module to be independently developed, tested, deployed and upgraded, greatly improving the development iteration efficiency and maintainability of the system.
[0038] The method of the above embodiment constructs a multi-modal interaction architecture coordinated by a central scheduling module and cooperated by multiple independent function agents, realizes high modularity of the system and low coupling degree between components, and improves the scalability, maintainability and development iteration efficiency of the system. In the multi-modal interaction process with the user, the central scheduling model understands the user request information, disassembles the user request information into multiple sub-tasks, and distributes the multiple sub-tasks to corresponding agents for processing, effectively realizing the processing of sub-tasks of different modalities, and achieving the effect of efficient and accurate multi-modal interaction with the user.
[0039] In some embodiments, according to the request information, the sub-tasks of the corresponding modalities are sent to each agent in the multiple agents, including: According to the request information, a plurality of sub-tasks of a plurality of modalities corresponding to the request information are generated, and an execution plan of the plurality of sub-tasks of the plurality of modalities is determined; According to the execution plan of the plurality of sub-tasks of the plurality of modalities, the sub-tasks of the corresponding modalities are sent to each agent in the multiple agents.
[0040] Specifically, in the embodiment of the present application, after the central scheduling model disassembles the user request information into multiple sub-tasks based on the understanding of the user request information, in order to realize efficient and accurate execution of the multiple sub-tasks of the multiple modalities, an execution plan of the multiple sub-tasks of the multiple modalities needs to be made. Optionally, the execution plan of the multiple sub-tasks of the multiple modalities includes at least one of the following: execution mode of the multiple sub-tasks of the multiple modalities, dependency relationship and execution order between the multiple sub-tasks of the multiple modalities. That is, by making the execution plan of the multiple sub-tasks of the multiple modalities, it can be accurately indicated which sub-tasks are dependent on each other and need to be executed in series, and which sub-tasks are independent of each other and can be executed in parallel, thereby greatly improving the execution efficiency of the multiple sub-tasks of the multiple modalities and improving the perception of multi-modal interaction with the user.
[0041] For example, the specific process of multi-modal interaction with the user in the embodiment of the present application is as follows: a. The user issues a query that may contain multiple intentions or involve multiple modal requirements through a user interaction interface.
[0042] b. The central scheduling module receives the query, parses the query, and identifies the core intention of the user and all related sub-tasks.
[0043] c. Task planning generation and optimization: the planning and decision unit of the central scheduling module generates a structured task execution plan based on the results of task decomposition. The plan describes in detail: Serial Task Sequence: Defines a sequence of Agent invocations that must be executed one after another in a specific order, where the input of a subsequent task may depend on the output of a previous task.
[0044] Parallel Task Group: Identifies multiple Agent invocations that can be executed simultaneously or concurrently, with no direct execution order dependency between these tasks.
[0045] Conditional Branching and Logic Control: May contain conditional branches based on the output of certain Agents, determining which branch of the execution plan to follow next.
[0046] Data Flow Definition: Clearly outlines the source of input data required for each Agent invocation (e.g., from user original input, output of a previous Agent, resource identifiers in the session context) and the intended use or delivery target of its output results.
[0047] d. Agent Scheduling and Execution: The Agent scheduling and execution engine of the central scheduling module is responsible for scheduling and executing functional Agents according to the generated task execution plan: For Serial Tasks: The scheduling engine strictly follows the order defined in the plan, invoking the corresponding Agents one after another. It waits for the previous Agent to complete its execution and return the results before passing the results (or a part of them) as input to the next Agent and initiating its invocation.
[0048] For Parallel Task Groups: The scheduling engine simultaneously (or through internal mechanisms like thread pools, task queues, etc.) issues invocation requests to all Agents within the group that are designated for parallel execution. These Agents will handle their respective tasks concurrently. The scheduling engine asynchronously monitors the execution status of these parallel tasks.
[0049] e. Dynamic Planning and Adaptive Feedback Handling: At any stage of task execution (whether serial or parallel tasks), if an Agent's execution returns an error message, a failure status, or its output results trigger a pre-set re-planning condition (e.g., results do not meet expectations, need for user further clarification, resources unavailable, etc.), the dynamic re-planning unit of the central scheduling module intervenes: It may pause or cancel the subsequent related pending tasks in the current plan.
[0050] According to the specific feedback information and the preset strategy, the current execution plan is modified. This can include replacing the failed Agent with a backup Agent, adjusting the calling parameters and retrying, adding new compensatory tasks (such as requesting additional information from the user), or in parallel tasks, if a branch fails, according to the strategy, it is decided whether to abort the entire parallel group, try to complete the task of the failed branch in other ways, or continue to execute the remaining part by ignoring the branch result.
[0051] f. Result aggregation and integration: For a group of tasks executed in parallel, the central scheduling module needs to set up a "convergence point" or "barrier" to wait for all parallel branches in the group to complete execution (or reach the timeout / failure termination condition) before collecting and integrating their respective output results. After that, the system will continue to process the subsequent serial tasks according to the execution plan, or integrate the final results.
[0052] g. Finally, the central scheduling module combines all the results generated by the relevant Agents after execution, after necessary integration and formatting, into a unified, multi-modal response, and presents it to the user through the user interaction interface.
[0053] That is, the central scheduling module in the embodiments of the present application has strong dynamic task planning and feedback adaptation capabilities, supporting complex serial and parallel task flow processing. Not only can it perform initial task planning, but more importantly, it can explicitly generate and execute complex Agent calls containing serial and parallel. This enables the system to effectively manage complex execution dependency relationships and data flow between multiple Agents. At the same time, the system can dynamically adjust and optimize the execution path according to the intermediate results and feedback information returned by each Agent during execution (including the success or failure status of individual branches in parallel tasks), thereby enhancing the robustness and intelligence of the system when facing complex and variable user demands and abnormal situations.
[0054] The method of the above embodiments, the execution plan of the plurality of sub-tasks of the plurality of modalities formulated by the central scheduling module includes the execution mode of the plurality of sub-tasks of the plurality of modalities, the dependency relationship and execution order between the plurality of sub-tasks of the plurality of modalities, and the data flow information between the plurality of sub-tasks of the plurality of modalities, which realizes dynamic adjustment and optimization of the execution path of the plurality of sub-tasks of the plurality of modalities, enhances the robustness and intelligence of the system when facing complex and variable user demands and abnormal situations, improves the execution efficiency of the plurality of sub-tasks of the plurality of modalities, and improves the perception of multi-modal interaction with the user.
[0055] In some embodiments, the multi-modal interaction method further includes: Obtaining target identifiers of various resources required by each sub-task execution from the session context and resource management module; sending the target identifiers to the respective agents.
[0056] Specifically, the session context and resource management module in the embodiments of the present application is used to persist or temporarily store the user and system interaction history of the multi-turn dialogue, monitor the filenames, URLs or other forms of external multimedia resources mentioned or uploaded by the user in the dialogue. When such resources are detected, they are securely cached (which can be stored in the local file system, object storage service or database), and a unique, short sequence number or identifier (for example, `resource_id_001`, `user_file_abc.pdf_ref`, `img_seq_05`) is assigned to them. This identifier is mapped to the storage location or access method of the actual resource. When the central scheduling module sends the sub-tasks of the corresponding modal to each agent in the plurality of agents according to the execution plan of the plurality of sub-tasks of the plurality of modal, the sequence number / identifier of the resource required for executing the sub-task is also transmitted to the Agent, instead of the long complete filename, file path or original data itself. When the Agent executes, it can request the resource manager to obtain the actual resource content or access permission through the identifier.
[0057] It should be noted that this mechanism significantly simplifies the parameter passing process when the Agent is called, avoids the failure of calling due to the long filename, the inclusion of special characters, path errors and the like, and thus improves the stability and efficiency of API calling. At the same time, it makes the resource reference in the dialogue context more clear, concise and secure.
[0058] The method of the above embodiments, by obtaining the target identifiers of various resources required for executing each sub-task, and sending the target identifiers to the respective agents, instead of the long complete filename, file path or original data itself, thus simplifies the parameter passing process when the Agent is called, avoids the failure of calling due to the long filename, the inclusion of special characters, path errors and the like, and improves the stability and efficiency of API calling.
[0059] The embodiments of the present application also provide a multi-modal interaction method, applied to any agent in a multi-modal interaction system, the agent being used to process sub-tasks of different modal, and the multi-modal interaction method comprising: receiving the sub-task of the corresponding modal of the agent sent by the central scheduling module; executing the sub-task of the corresponding modal, and sending the execution result of the sub-task of the corresponding modal to the central scheduling module.
[0060] Specifically, in the embodiments of the present application, the central scheduling module, based on the understanding of the user request information, disassembles the user request information into multiple sub-tasks, and distributes the multiple sub-tasks to corresponding agents for processing. Optionally, different agents are used to process sub-tasks of different modalities, and each agent, after receiving the sub-tasks of the corresponding modality of the agent sent by the central scheduling module, can execute the sub-tasks of the corresponding modality, and send the execution results of the sub-tasks of the corresponding modality to the central scheduling module. After the central scheduling module receives the execution results of the sub-tasks of each modality sent by each agent, it can combine the sub-task execution results generated after the execution of all related agents Agent, after necessary integration and formatting, into a unified, multi-modal response, and present it to the user through the user interaction interface, thereby efficiently and accurately realizing multi-modal interaction with the user.
[0061] The method of the above embodiments constructs a multi-modal interaction architecture that is uniformly coordinated by a central scheduling module and cooperated by multiple independent functional agents, realizes high modularity of the system and low coupling degree between components, and improves the scalability, maintainability and development iteration efficiency of the system. In the process of multi-modal interaction with the user, each agent, after receiving the sub-tasks of the corresponding modality of the agent sent by the central scheduling module, can execute the sub-tasks of the corresponding modality, and send the execution results of the sub-tasks of the corresponding modality to the central scheduling module, thereby effectively realizing the processing of sub-tasks of different modalities, and achieving the effect of efficient and accurate multi-modal interaction with the user.
[0062] In some embodiments, the multi-modal interaction method further comprises: receiving target identifiers of various resources required for execution of the sub-tasks sent by the central scheduling module; According to the target identifiers, obtaining the various resources required for execution of the sub-tasks from the session context and resource management module, and executing the sub-tasks of the corresponding modality of the agent sent by the central scheduling module.
[0063] Specifically, the session context and resource management module in this embodiment is used to persistently or temporarily store the user's interaction history with the system during multi-turn dialogues, and to monitor filenames, URLs, or other forms of external multimedia resources mentioned or uploaded by the user in the dialogue. When such resources are detected, they are securely cached and assigned a unique, short sequence number or identifier. This identifier is mapped to the actual storage location or access method of the resource. After the central scheduling module sends the corresponding modal subtasks to each agent in the multiple agents according to the execution plan of multiple subtasks of multiple modalities, it also passes the sequence number / identifier of the resource required to execute the subtask to the Agent, instead of a lengthy complete filename, file path, or the original data itself. When the Agent executes, it can use this identifier to request the actual resource content or access permissions from the resource manager.
[0064] The method described above simplifies the parameter passing process during Agent calls by obtaining the target identifiers of various resources required for the execution of each subtask, rather than lengthy complete filenames, file paths, or the original data itself. This avoids call failures caused by excessively long filenames, special characters, or incorrect paths, thereby improving the stability and efficiency of API calls.
[0065] For example, such as Figure 2 and Figure 3 As shown in the figure, this application provides a structural diagram of a multimodal interaction system and a multimodal interaction method, as detailed below: 1. Central Dispatch Model: Core Responsibilities: Receive multimodal queries from users (which may include text, voice, images, etc.).
[0066] By combining the context of the conversation and using natural language understanding technology, we can conduct in-depth analysis of user queries and accurately identify user intent.
[0067] Complex user requests are decomposed into a series of atomic or combined subtasks that can be executed by the functional agent.
[0068] Generate a structured task execution plan: This plan clearly defines each subtask (i.e., a call to a specific functional agent), the execution dependencies between subtasks (which determine which tasks must be executed sequentially), which subtasks can be executed concurrently (i.e., called in parallel), and how data flows and is transferred between different agents.
[0069] According to the generated task execution plan, one or more Agents in the functional Agent pool are selected and instantiated.
[0070] Call instructions are issued to the selected Agents, and necessary input parameters (including resource identifiers obtained through the resource manager) are passed.
[0071] Collaboration and data flow between Agents are managed to ensure that data is correctly passed between different Agents as planned.
[0072] The execution status and return results of the Agents are monitored.
[0073] Based on the intermediate results or feedback information returned during Agent execution (such as success, failure, error code, need for additional information, etc.), the effectiveness of the current execution plan is dynamically evaluated, and if necessary, the existing plan is adjusted (such as replacing failed Agents, modifying parameters, changing the execution order) or a new task branch is generated.
[0074] After all related sub-tasks are executed, the output results of each Agent are integrated to form a final, unified multi-modal response.
[0075] The final response is presented to the user.
[0076] That is, in the embodiments of the present application, the high modularity and low coupling of the system are achieved through the central scheduling-multi-Agent collaboration architecture, and various modal processing capabilities including LLM are encapsulated as independent, pluggable functional Agents, which are uniformly, flexibly, and dynamically planned and scheduled by a dedicated central scheduling module. This architecture allows each functional module to be independently developed, tested, deployed, and upgraded, greatly improving the development iteration efficiency and maintainability of the system. Moreover, the multi-modal interaction system in the embodiments of the present application has strong dynamic task planning and feedback adaptation capabilities, supporting complex serial and parallel task flow processing. The central scheduling module not only can perform initial task planning, but more importantly, it can explicitly generate and execute complex Agent calls containing serial and parallel. This enables the system to effectively manage complex execution dependency relationships and data flow between multiple Agents. By effectively parallelizing non-dependent sub-tasks, the overall response time of complex multi-modal requests can be significantly shortened, significantly improving the processing efficiency and system throughput of complex multi-modal tasks, and improving the overall processing request capability of the system, thereby improving user experience. At the same time, the system can also dynamically adjust and optimize the execution path according to the intermediate results and feedback information returned by each Agent during execution (including the success or failure status of individual branches in parallel tasks), thereby enhancing the robustness and intelligence of the system in the face of complex and variable user demands and abnormal situations.
[0077] Technical Features: This module focuses on advanced reasoning, planning, and scheduling logic, rather than the specific generation of content in a particular modality. It can be a specially trained model with strong planning capabilities or a decision engine based on rules and heuristic algorithms. The output of the planning can explicitly support complex control flows such as serial, parallel, conditional branching, and loops.
[0078] 2. Functional Agent Pool: Composition: It contains a series of modular functional agents with standardized interfaces. Each agent is designed to be responsible for a specific type of modality processing task or general computing task. Example agents include: Language Model Agent (LLM Agent): Encapsulates advanced LLMs (such as GPT series, Llama series, etc.) and is specifically responsible for text understanding, text generation, dialogue management, knowledge question answering, text summarization, text creation, code generation, and other language-related core tasks.
[0079] Visual Language Model Agent (VLM Agent): Encapsulates advanced visual language models (such as GPT-4V, LLaVA, CogVLM, etc.) that can receive images and text as joint input and perform understanding, reasoning, and text generation based on these inputs. For example, it can answer questions about image content, describe images, generate stories based on image and text instructions, or make referential descriptions of specific areas in the image.
[0080] Image Generation Agent (T2I Agent): Encapsulates text-to-image models (such as Stable Diffusion, DALL-E, Midjourney, etc.) that are responsible for generating images based on text descriptions or other inputs.
[0081] Speech Recognition Agent (ASR Agent): Encapsulates speech-to-text services (such as Whisper API, Google Cloud Speech-to-Text, etc.) that are responsible for converting user speech input into text.
[0082] Speech Synthesis Agent (TTS Agent): Encapsulates text-to-speech engines (such as ElevenLabs, Microsoft Azure TTS, etc.) that are responsible for converting system-generated text into natural and fluent speech output.
[0083] Other possible Agents: Video processing Agents (e.g. content analysis, editing, generation), data analysis and visualization Agents, knowledge base retrieval Agents, external API calling Agents (for integration with third-party services), code execution Agents (for safely executing user-provided code snippets), etc.
[0084] Optionally, the Agent-based implementation of LLM in the embodiments of the present application achieves role separation, reducing the training and maintenance costs of LLM: transforming LLM from a "jack-of-all-trades" that tries to handle all tasks into a specialized Agent that focuses on its core strengths (text understanding, generation, and reasoning). This avoids the need to force LLM to adapt to all modalities of generation capabilities and complex scheduling logic, significantly reducing the training complexity and maintenance costs of LLM itself, and making it easier to maintain the stability and high performance of its core text processing capabilities. That is, the embodiments of the present application achieve the role repositioning of language models, transforming large language models (LLM) from traditional central processing units into members of a pool of functional Agents, focusing on their core text understanding, generation, and reasoning capabilities. This role separation simplifies the training and optimization requirements of LLM, reducing the cost and complexity of integrating it into complex multi-modal systems.
[0085] Technical features: Each Agent can be independently developed, tested, deployed, optimized, and versioned without affecting other components of the system. Agents communicate with each other through predefined interfaces that specify the format of input parameters, the structure of output results, and possible error codes. The design of Agents should support asynchronous invocation patterns to facilitate efficient parallel task execution by the central scheduling module.
[0086] 3. Session Context and Resource Manager: Core responsibilities: Persist or temporarily store the user-system interaction history of multiple rounds of conversations.
[0087] Monitor the filenames, URLs, or other forms of external multimedia resources mentioned or uploaded by the user in the conversation.
[0088] When such resources are detected, they are securely cached (which can be stored in the local file system, object storage service, or database) and assigned a unique, short sequence number or identifier (e.g., `resource_id_001`, `user_file_abc.pdf_ref`, `img_seq_05`). This identifier maps to the storage location or access method of the actual resource.
[0089] When the central scheduling module instructs a certain functional Agent to process these cached resources, it passes the serial number / identifier of the resource to the Agent instead of the lengthy full file name, file path, or original data itself. The Agent can request the actual resource content or access permission from the resource manager through the identifier when executing.
[0090] It should be noted that the conversation resource management mechanism based on serial numbers / identifiers in the embodiments of the present application improves the simplicity and robustness of the call. The files or external multimedia resources involved in the conversation history are cached and referenced by unique and short serial numbers or identifiers. This way simplifies the parameter passing when the Agent calls, avoids problems that may be caused by passing lengthy and error-prone file paths or full file names, thereby improving the stability and overall efficiency of API calls.
[0091] Technical advantages: This mechanism significantly simplifies the parameter passing process when the Agent calls, avoids call failures caused by long file names, special characters, path errors, etc., thereby improving the stability and efficiency of API calls. At the same time, it makes the resource reference in the conversation context more clear, concise and secure.
[0092] In the above multi-modal interaction system, different modal processing capabilities (such as text understanding and generation, image generation, speech recognition, speech synthesis, etc.) are encapsulated as independent "functional Agents" with standard interfaces. It is worth noting that the language model (LLM) itself is also regarded as a functional Agent that specializes in processing text-related tasks. The system sets up an independent "central scheduling module", which is responsible for receiving and understanding the user's global intent, generating a task plan containing serial and / or parallel execution steps, selecting and scheduling appropriate functional Agents to execute corresponding sub-tasks according to the plan, and dynamically adjusting the existing plan or generating new plan branches according to the Agent's execution feedback (such as success, failure, partial results), thereby effectively overcoming the shortcomings of existing solutions in flexibility, iteration efficiency, cost control, dynamic planning ability, and complex serial and parallel multi-tool collaborative processing ability.
[0093] The multi-modal interaction device provided by the present application is described below. The multi-modal interaction device described below can be correspondingly referred to the multi-modal interaction method described above. The multi-modal interaction device of the embodiments of the present application, as shown in Figure 4 includes: The acquisition module 410 is configured to acquire the request information of the user. The processing module 420 is configured to send a sub-task of a corresponding modality to each intelligent agent in the plurality of intelligent agents according to the request information. Each intelligent agent in the plurality of intelligent agents is configured to process a sub-task of a different modality. The receiving module 430 is configured to receive the execution results of the sub-tasks of different modalities sent by the plurality of agents.
[0094] Figure 5 An example of an entity structure diagram of an electronic device is shown, which can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 can communicate with each other through the communications bus 540. The processor 510 can invoke the logical instructions in the memory 530 to execute a multi-modal interaction method, which includes: obtaining request information of a user; sending a sub-task of a corresponding modality to each agent in a plurality of agents according to the request information; each agent in the plurality of agents is configured to process a sub-task of a different modality; receiving the execution results of the sub-tasks of different modalities sent by the plurality of agents; or, receiving the sub-task of the corresponding modality of the agent sent by the center scheduling module; executing the sub-task of the corresponding modality, and sending the execution result of the sub-task of the corresponding modality to the center scheduling module.
[0095] In addition, the logical instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.
[0096] In another aspect, the present application also provides a computer program product comprising a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the multi-modal interaction method provided by any of the above methods, the method comprising: obtaining request information of a user; sending a subtask of a corresponding modality to each of a plurality of agents according to the request information; each of the plurality of agents is configured to process a subtask of a different modality; receiving an execution result of the subtask of the different modality sent by the plurality of agents; or, receiving a subtask of a corresponding modality of an agent sent by a central scheduling module; executing the subtask of the corresponding modality, and sending the execution result of the subtask of the corresponding modality to the central scheduling module.
[0097] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the multi-modal interaction method provided by any of the above methods, the method comprising: obtaining request information of a user; sending a subtask of a corresponding modality to each of a plurality of agents according to the request information; each of the plurality of agents is configured to process a subtask of a different modality; receiving an execution result of the subtask of the different modality sent by the plurality of agents; or, receiving a subtask of a corresponding modality of an agent sent by a central scheduling module; executing the subtask of the corresponding modality, and sending the execution result of the subtask of the corresponding modality to the central scheduling module.
[0098] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.
[0099] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary general hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in terms of the contribution to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform the methods described in the various embodiments or some parts of the embodiments.
[0100] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A multimodal interaction method, characterized in that, A central scheduling module applied in multimodal interaction systems includes: Obtain the user's request information; Based on the request information, sub-tasks of the corresponding modality are sent to each of the multiple intelligent agents; each of the multiple intelligent agents is used to process sub-tasks of different modalities. Receive the execution results of subtasks of different modalities sent by the multiple intelligent agents.
2. The multimodal interaction method according to claim 1, characterized in that, The step of sending sub-tasks of corresponding modalities to each of the multiple agents according to the request information includes: Based on the request information, generate multiple sub-tasks of multiple modalities corresponding to the request information, and determine the execution plan of the multiple sub-tasks of multiple modalities; Based on the execution plan of the multiple subtasks of the multiple modalities, the subtasks of the corresponding modalities are sent to each of the multiple agents.
3. The multimodal interaction method according to claim 2, characterized in that, The execution plan for the multiple subtasks of the multiple modalities includes at least one of the following: The execution methods of the multiple subtasks of the multiple modalities, the dependencies and execution order between the multiple subtasks of the multiple modalities, and the data flow information between the multiple subtasks of the multiple modalities.
4. The multimodal interaction method according to claim 3, characterized in that, The method further includes: Obtain the target identifiers of various resources required for the execution of each subtask from the session context and resource management module; The target identifier is sent to each of the intelligent agents.
5. A multimodal interaction method, characterized in that, The method is applied to any agent in a multimodal interaction system, the agent being used to handle sub-tasks of different modalities; the method includes: The receiving center scheduling module sends the corresponding modality subtask of the agent; Execute the subtask of the corresponding modality and send the execution result of the subtask of the corresponding modality to the central scheduling module.
6. The multimodal interaction method according to claim 5, characterized in that, The method further includes: Receive the target identifiers of various resources required for the execution of the subtask sent by the central scheduling module; Based on the target identifier, obtain the various resources required for the execution of the subtask from the session context and resource management module, and execute the subtask of the corresponding modality of the agent sent by the central scheduling module.
7. A multimodal interaction device, characterized in that, include: The acquisition module is used to acquire user request information; The processing module is used to send subtasks of the corresponding modality to each of the multiple intelligent agents according to the request information; Each of the multiple agents is used to process subtasks of different modalities; The receiving module is used to receive the execution results of sub-tasks of different modalities sent by the multiple intelligent agents.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal interaction method as described in any one of claims 1 to 4 or the multimodal interaction method as described in any one of claims 5 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal interaction method as described in any one of claims 1 to 4 or the multimodal interaction method as described in any one of claims 5 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal interaction method as described in any one of claims 1 to 4 or the multimodal interaction method as described in any one of claims 5 to 6.