Intelligent agent communication interaction method, electronic device, and computer-readable storage medium

Abstract

This application discloses an intelligent agent communication interaction method, an electronic device, and a computer-readable storage medium, relating to the fields of artificial intelligence and large-scale model technology. The method includes: generating a trigger signal based on currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; determining the multimodal conversation content using the trigger signal and pre-stored historical dialogue data; performing conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed; distributing the sequence of actions to be executed to multiple intelligent agents for execution to obtain execution results; and feeding back the execution results to the target object. This application solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple intelligent agents, failing to meet diverse scenario requirements and personalized needs.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and large model technology, and more specifically, to an intelligent agent communication and interaction method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of large-scale models and multimodal technologies, the application of artificial intelligence (AI) hardware is ushering in a new wave of popularity. Products such as smart cockpits and wearable devices have significantly improved the interactive experience by integrating multimodal large-scale models.

[0003] Currently, traditional multimodal interaction systems mostly employ end-to-end speech modeling schemes, achieving a certain level of multimodal understanding and interaction by jointly pre-training data from multiple modalities. However, end-to-end speech modeling schemes generally lack tool usage capabilities and deep customization functions, making it difficult to cope with complex and ever-changing scenario requirements and provide personalized services. Furthermore, while full-duplex speech modeling schemes have been proposed, improving real-time interactivity, they lack the ability to process other modal information such as video, limiting their application breadth in multimodal environments and failing to meet diverse scenario and user needs.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an intelligent agent communication and interaction method, an electronic device, and a computer-readable storage medium to at least solve the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple intelligent agents and cannot meet diverse scenario requirements and personalized needs.

[0006] According to one aspect of the embodiments of this application, an intelligent agent communication interaction method is provided, comprising: generating a trigger signal based on currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; determining multimodal conversation content using the trigger signal and pre-stored historical dialogue data; performing conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed; assigning the sequence of actions to be executed to multiple intelligent agents for execution to obtain execution results; and feeding back the execution results to the target object.

[0007] According to another aspect of the embodiments of this application, an intelligent agent communication interaction method is also provided, comprising: generating a trigger signal based on a currently input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction mode of the target object; determining the multimodal merchant recommendation session content using the trigger signal and pre-stored historical merchant recommendation dialogue data; performing session analysis on the multimodal merchant recommendation session content to obtain a merchant recommendation action sequence; distributing the merchant recommendation action sequence to multiple intelligent agents for execution to obtain a merchant recommendation execution result; and feeding back the merchant recommendation execution result to the target object.

[0008] According to another aspect of the embodiments of this application, an intelligent agent communication interaction method is also provided, comprising: obtaining an intelligent agent communication interaction request through a first application programming interface, wherein the request data carried in the intelligent agent communication interaction request includes: multimodal information, the multimodal information being determined according to the multimodal interaction mode of the target object; and returning an intelligent agent communication interaction response through a second application programming interface, wherein the response data carried in the intelligent agent communication interaction response includes: execution result, the execution result being obtained according to any of the above-mentioned intelligent agent communication interaction methods.

[0009] According to another aspect of the embodiments of this application, an intelligent agent communication interaction method is also provided, comprising: acquiring a currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, the multimodal information being determined according to the multimodal interaction mode of the target object; responding to the intelligent agent communication interaction dialogue request, returning an intelligent agent communication interaction dialogue response, wherein the information carried in the intelligent agent communication interaction dialogue response includes: execution result, the execution result being obtained according to any of the above-mentioned intelligent agent communication interaction methods; and displaying the execution result in a graphical user interface.

[0010] According to another aspect of the embodiments of this application, an intelligent agent communication interaction method is also provided, including: responding to an input command applied to an operation interface and displaying multimodal information on the operation interface; responding to a processing command applied to the operation interface and displaying an execution result on the operation interface; wherein the execution result is obtained according to any of the above-mentioned intelligent agent communication interaction methods.

[0011] According to another aspect of the embodiments of this application, an intelligent agent communication interaction system is also provided, comprising: a perception layer, used to generate a trigger signal based on currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; a control layer, used to determine the multimodal conversation content using the trigger signal and pre-stored historical dialogue data; a planning layer, used to perform conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed; an execution layer, used to assign the sequence of actions to be executed to multiple intelligent agents for execution to obtain execution results; and the planning layer is further used to feed back the execution results to the target object.

[0012] According to another aspect of the embodiments of this application, an intelligent agent communication and interaction system is also provided, including: a client, used to send multimodal information, wherein the multimodal information is determined based on the multimodal interaction mode of the target object; a server, connected to the client, used to generate a trigger signal based on the multimodal information, use the trigger signal and pre-stored historical dialogue data to determine the multimodal conversation content, perform conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed, and assign the sequence of actions to be executed to multiple intelligent agents for execution to obtain an execution result; the client is also used to output the execution result.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to execute any of the above-described intelligent agent communication interaction methods.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described intelligent agent communication and interaction methods.

[0016] In this embodiment, a trigger signal is generated based on the multimodal information of the current input. The multimodal information is determined according to the multimodal interaction mode of the target object. Then, the multimodal conversation content is determined by using the trigger signal and pre-stored historical dialogue data. Next, the multimodal conversation content is analyzed to obtain a sequence of actions to be executed. Then, the sequence of actions to be executed is assigned to multiple agents for execution to obtain the execution result. Finally, the execution result is fed back to the target object. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient and personalized multimodal interaction for the needs of multiple scenarios. This solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration between multiple agents and cannot meet diverse scenario needs and personalized needs.

[0017] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of an intelligent agent communication and interaction method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the operation of a multimodal multi-agent interaction framework according to an embodiment of this application;

[0022] Figure 4 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application;

[0023] Figure 5 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application;

[0024] Figure 6 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application;

[0025] Figure 7 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of the structure of an intelligent agent communication and interaction system according to an embodiment of this application;

[0027] Figure 9 This is a schematic diagram of the structure of an intelligent agent communication and interaction system according to an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the structure of an intelligent agent communication and interaction device according to an embodiment of this application;

[0029] Figure 11 This is a schematic diagram of the structure of another intelligent agent communication and interaction device according to an embodiment of this application;

[0030] Figure 12 This is a schematic diagram of the structure of another intelligent agent communication and interaction device according to an embodiment of this application;

[0031] Figure 13 This is a schematic diagram of the structure of another intelligent agent communication and interaction device according to an embodiment of this application;

[0032] Figure 14 This is a schematic diagram of the structure of another intelligent agent communication and interaction device according to an embodiment of this application;

[0033] Figure 15 This is a structural block diagram of a computing device according to an embodiment of this application;

[0034] Figure 16 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] The technical solution provided in this application is mainly implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models can also be called foundation models. They are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability, such as Large Language Models (LLMs) and multi-modal pre-training models.

[0038] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely used in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, the example of using a large model for agent communication interaction in a multimodal interaction exploration scenario is used for explanation.

[0039] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0040] Multilevel memory: The memory management model of a dialogue system, including scene memory, dialogue memory, and long-term memory.

[0041] Multi-modal: By integrating multiple information types such as text, images, voice, and video, it enables more natural and comprehensive information interaction and understanding.

[0042] Multi-Agent: A collaborative system composed of multiple autonomous intelligent agents that improves overall efficiency through distributed decision-making, task allocation, and cooperation.

[0043] Perception layer: The basic module for the system to receive external information, responsible for data acquisition, preprocessing and feature extraction (such as visual recognition and speech-to-text).

[0044] Action planning: Generates a sequence of actions for the agent based on the environmental state and user instructions.

[0045] Interactive system: A framework that supports two-way information exchange between humans and machines or intelligent agents, including input parsing, feedback generation, and execution feedback closed loop.

[0046] Enhanced multimodal search: Integrating cross-modal semantic matching technology (such as text-image joint retrieval) to improve the accuracy and speed of information retrieval in complex scenarios.

[0047] Voice Activity Detection (VAD): A technique for identifying speech and non-speech segments in an audio signal, used for noise reduction, speech endpoint detection, and resource optimization.

[0048] Large-scale models need to understand the world and move closer to Artificial General Intelligence (AGI). This requires breaking through the boundaries of text-only models. In recent years, numerous studies have attempted to jointly pre-train text with other modalities to achieve multimodal understanding. Simultaneously, advancements in large-scale and multimodal models have fueled a new wave of AI hardware applications, with manufacturers of smart cockpits and smart speakers releasing large-scale models and upgrading their performance.

[0049] Currently, most multimodal interaction systems adopt end-to-end speech modeling solutions. These solutions lack tooling capabilities and have limited customizability, making them difficult to scale realistically to business (B2B) customer scenarios. They are more often used as toys for casual conversation and companionship. Furthermore, these solutions lack long-term memory and user personalization capabilities. An end-to-end full-duplex speech modeling solution has also been proposed, where the model directly receives streaming speech input and generates streaming speech output. This solution is non-interventional, does not support video input, and faces serious issues with language model degradation. In addition, traditional speech interaction frameworks are rule-based and lack generalization capabilities, failing to meet the performance requirements of end users.

[0050] The multimodal interaction systems of related technologies have the following drawbacks.

[0051] Defect 1: The end-to-end speech modeling solution fails to effectively integrate external tools or knowledge bases, limiting its functional scalability and practicality in real-world scenarios. Furthermore, the model has low customizability, making it difficult to adapt to the diverse needs of B2B clients and extending to more business scenarios. In addition, the lack of an effective long-term memory mechanism means that user preferences and historical information cannot be personalized during interactions, reducing the continuity and satisfaction of the user experience.

[0052] Defect 2: The end-to-end duplex speech modeling solution only uses duplex speech modeling and does not support video input, which limits the comprehensiveness of scene understanding and information interaction, and faces the problem of insufficient language model understanding ability.

[0053] Defect 3: Interaction frameworks that rely on traditional rules have limited generalization ability when dealing with unknown or complex scenarios, making it difficult to meet the growing intelligent needs of end users.

[0054] To address the aforementioned deficiencies, no effective solution has been proposed prior to this application.

[0055] According to an embodiment of this application, an intelligent agent communication interaction method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0056] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the method provided in this application embodiment can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1In the application scenario shown, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to access the large model, thereby implementing the method provided in this embodiment.

[0057] In this embodiment, the system consisting of a client device and a server can perform the following steps: the client device acquires the currently input multimodal information and sends the multimodal information to the server; the server generates a trigger signal based on the currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; the multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data; the multimodal conversation content is analyzed to obtain a sequence of actions to be executed; the sequence of actions to be executed is assigned to multiple intelligent agents for execution, the execution result is obtained, and the execution result is fed back to the client device (i.e., the client device used by the target object, or the entity device for multimodal interaction). It should be noted that, provided that the operating resources of the client device can meet the deployment and operation conditions of the large model, this embodiment can be performed on the client device.

[0058] It should be noted that with the rapid development of high-performance computing units, the methods provided in this application embodiment can also be applied to model-in-machine systems in other application scenarios. In one optional embodiment, the model-in-machine system has multiple built-in models, and users can select one model to adjust as needed to obtain their own model. The high-performance computing unit built into the model-in-machine system can then directly call the adjusted model to execute the methods provided in this application embodiment. In another optional embodiment, the large model-in-machine system has a pre-trained model built-in, and the high-performance computing unit built into the model-in-machine system can then directly call that model to execute the methods provided in this application embodiment.

[0059] Furthermore, when users need to train their own models, they can upload their own datasets via the client. These datasets are then sent to the server, allowing the server to adjust the pre-trained model using the dataset to obtain the user's customized model, which can then be deployed to the production environment. To facilitate users' model adjustment needs, the server provides complete adjustment tools, development frameworks, and processes, supporting multiple adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0060] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for intelligent agent communication and interaction is shown. Figure 2 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application. Figure 2 As shown, the method may include the following steps:

[0061] Step S21: Generate a trigger signal based on the currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object;

[0062] Step S22: Determine the multimodal conversation content using the trigger signal and pre-stored historical dialogue data;

[0063] Step S23: Perform conversation analysis on the multimodal conversation content to obtain the sequence of actions to be executed;

[0064] Step S24: Distribute the sequence of actions to be executed to multiple agents for execution, and obtain the execution results;

[0065] Step S25: Feedback the execution result to the target object.

[0066] In this embodiment, the currently input multimodal information can be understood as the multimodal information input by the target object. The target object can be understood as a user or entity that interacts with the smart device in a multimodal manner. It is understood that the smart device may be equipped with the multimodal interaction system proposed in this application. For example, the target object can be a person using a client device, such as a person using a smartphone, smart wearable device, smart cockpit, or other smart hardware. It can also be any object that can communicate with the smart device in multiple ways, such as voice, image, and text. For example, sensors in home automation devices (such as temperature and light sensors) can also be part of the target object, sending multimodal data (such as voice broadcast of temperature values ​​or image display of light intensity) to the smart device. There are no limitations here.

[0067] Multimodal information refers to information that simultaneously includes multiple types, such as text, voice, and image information, without limitation. Multimodal information is determined based on the multimodal interaction methods of the target object. Multimodal interaction methods can be understood as combinations of various information types used by the target object to communicate with the smart device, such as interactions combining voice, text, images, video, or gestures, without limitation.

[0068] A trigger signal can be understood as an indication signal generated based on the analysis of multimodal information of the current input, enabling the smart device to respond promptly to the interactive behavior of the target object. In essence, a trigger signal represents the multimodal information of the current input. For example, a smart device can monitor and parse user input information in real time to generate a trigger signal. For instance, when meaningful user activity is detected, such as when the user begins to speak, the environment changes, or the user inputs information through the screen, the smart device generates a corresponding trigger signal.

[0069] In this application, generating a trigger signal based on the current input multimodal information can be understood as follows: the intelligent device first receives the current input multimodal information from the target object, and then generates a specific trigger signal based on the multimodal information. It can be seen that by integrating and analyzing multimodal information, the intelligent device can more accurately understand the user's intent, improve the accuracy of trigger events, and thus provide more intelligent and responsive services.

[0070] After generating a trigger signal, the multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data. Historical dialogue data can be understood as pre-stored records of interactions between the user and the smart device, used to build long-term memory and personalized experiences. For example, historical dialogue data includes, but is not limited to, text chat logs, voice dialogue transcripts, image feedback history, video communication information, and user behavior habits. It can be short-term or long-term historical dialogue data; no limitation is imposed here.

[0071] Multimodal conversation content can include multimodal information and the current conversation context determined based on historical dialogue data. It can also include user device information so that the multimodal interaction system can fully understand the user's needs and context, thereby providing more accurate and natural responses.

[0072] In this application, the multimodal conversation content is determined using trigger signals and pre-stored historical dialogue data. This can be understood as the smart device, based on the generated trigger signal, calling upon and integrating the user's historical dialogue data to generate multimodal conversation content, enabling the multimodal interaction system to more comprehensively understand the background and context of the current interaction. This allows for a more natural and fluid interaction, enhancing user trust and satisfaction with the multimodal interaction system.

[0073] After determining the multimodal conversation content, conversation analysis is performed on the content to obtain a sequence of actions to be executed. This sequence can be understood as a series of operation instructions, or an action list, used to respond to user requests. Specifically, it can be understood as including at least one operation instruction for an action to be executed. For example, semantic understanding, intent recognition, and scenario analysis can be performed on the multimodal conversation content to obtain the sequence of actions to be executed, ensuring that the multimodal interaction system can correctly interpret user needs and formulate corresponding action plans.

[0074] In this application, session analysis is performed on the multimodal session content to obtain a sequence of actions to be executed. This can be understood as the intelligent device performing in-depth analysis of the multimodal session content, transforming the user's intent and needs into a specific sequence of actions to be executed. This improves the accuracy and efficiency of the multimodal interaction system in executing tasks, avoids misunderstanding user intent, ensures service precision, and enhances the user experience.

[0075] After obtaining the sequence of actions to be executed, the sequence is distributed to multiple agents for execution, yielding the results. An agent can be understood as a software entity (intelligent agent) with autonomous decision-making and execution capabilities in a multimodal interaction system. Each agent typically focuses on processing a specific modality (such as speech, text, image, video, etc.) or a specific type of task (such as search, translation, visual recognition, etc.). Multiple agents can work independently as well as collaborate to complete complex and diverse user requests or tasks.

[0076] The execution result can be understood as the output generated after multiple intelligent agents complete the task, including response information to user instructions.

[0077] In this application, the sequence of actions to be executed is distributed to multiple intelligent agents for execution, and the execution result is obtained. This can be understood as the intelligent device intelligently assigning tasks in the sequence of actions to the most suitable intelligent agent or multiple intelligent agents for execution. Each intelligent agent handles the corresponding task according to its expertise, and returns the execution result after completing the operation. Therefore, the multimodal interaction system of this application allows multiple intelligent agents to process tasks in parallel, improving the processing capability and response speed in complex scenarios, while ensuring the accuracy and richness of task execution, and enhancing the intelligence and flexibility of the multimodal interaction system.

[0078] After obtaining the execution result, the result is fed back to the target object. This can be understood as the smart device sending the execution result back to the target object, thus completing one interaction loop. For example, the smart device can provide feedback through voice replies, image displays, or other forms of output to ensure that the user can understand the system response in a timely and intuitive manner; there are no restrictions on this.

[0079] Understandably, intelligent devices can stream all execution results back to the target object. This means that intelligent devices can continuously and dynamically return the execution results obtained by each agent to the user in the form of a real-time data stream. This ensures the real-time and continuous response of the multimodal interaction system. Users can instantly see or hear the processing results of the multimodal interaction system. The dynamic interaction mode improves the user experience, making the entire conversation smoother and more intuitive. Correspondingly, intelligent devices can also package all execution results into a static, complete output and send it to the user; this is not restricted here.

[0080] As can be seen, this application allows multimodal interaction systems to process multiple types of information simultaneously, thereby gaining a more comprehensive understanding of the user's intent and context, and providing responses that closely resemble natural language communication. Furthermore, by incorporating the user's historical dialogue behavior, the multimodal interaction system can reference personalized information during interaction, achieving more intelligent and personalized services, meeting individual needs, and enhancing the interactive intelligence and naturalness of the multimodal interaction system.

[0081] Furthermore, this application ensures the real-time and continuous response of the multimodal interaction system by providing real-time streaming feedback of execution results. Simultaneously, cross-modal multi-agent collaboration is achieved through the division of labor and cooperation among different agents, allowing for the dynamic scheduling of suitable agents to respond to different user scenario needs, thus meeting diverse scenario requirements. For example, switching between voice queries and visual feedback satisfies the diverse and complex interaction needs of users.

[0082] The intelligent agent communication and interaction method provided in this application embodiment can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0083] By employing the embodiments of this application, a trigger signal is generated based on the multimodal information of the current input. The multimodal information is determined according to the multimodal interaction mode of the target object. Then, the multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data. Next, the multimodal conversation content is analyzed to obtain a sequence of actions to be executed. The sequence of actions to be executed is then assigned to multiple agents for execution to obtain the execution result. Finally, the execution result is fed back to the target object. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios. This solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0084] In an optional embodiment, in step S21, a trigger signal is generated based on the currently input multimodal information, including the following method steps:

[0085] Step S211: Perceive and understand the currently input multimodal information to obtain an understanding result, wherein the understanding result is used to perceive the intention or action of the target object through multimodal information;

[0086] Step S212: Generate a trigger signal based on the understanding result.

[0087] It is understood that the smart device is equipped with the multimodal interaction system proposed in this application. The multi-agent interaction framework of the multimodal interaction system may include a perception layer. The perception layer serves as the window through which the entire multimodal interaction system interacts directly with the user. It is used to monitor and parse the user's input information in real time, and then generate trigger signals based on the parsing results.

[0088] In this embodiment, when generating a trigger signal based on the currently input multimodal information, the currently input multimodal information can be perceptually understood to obtain an understanding result, and then the trigger signal can be generated based on the understanding result. The perceptual understanding of the currently input multimodal information can be understood as the processing procedure of the perception layer, used to convert multimodal information into a form understandable by the multimodal interaction system, including recognizing, parsing, and understanding different multimodal information.

[0089] For example, the perception layer can use sound activity detection technology to detect whether the user has started speaking and whether speaking has ended; it can also use visual scene transition detection technology to detect whether the user is moving to a different environment; it can also use image recognition technology to analyze specific objects; and it can use other technologies to analyze text and other information. It is understood that the perception layer can use different detection results to generate trigger signals as needed, and this is not limited here.

[0090] The understanding result can be understood as the information obtained by the perception layer after perceiving and understanding multimodal information. It is used to perceive the intention or action of the target object through multimodal information. That is, the understanding result can contain a preliminary interpretation of the target object's intention or behavior. For example, the understanding result may indicate "the user is asking about the weather" or "the user picked up a book," without limitation.

[0091] In this application, the multimodal information input is perceived and understood. The resulting understanding can be understood as follows: the perception layer in the intelligent device first receives multimodal information from the target object, then analyzes and understands this information, and finally integrates it into a single understanding result. This understanding result reflects the target object's intention or ongoing action. Therefore, by perceiving and understanding multimodal information, multimodal interaction can more comprehensively capture user needs, improve the accuracy and response speed of the interaction, and provide precise contextual information for subsequent intelligent processing.

[0092] The perception layer then generates one or more trigger signals based on the obtained understanding results. For example, if the understanding results indicate that the user is asking about the weather, the perception layer will generate a trigger signal related to the weather query. If the user is describing a visual scene, the perception layer will generate a trigger signal related to image recognition or visual search, and so on.

[0093] It is understandable that the multi-agent interaction framework of a multimodal interaction system can also include a control layer, which receives trigger signals from the perception layer. This can be understood as the perception layer sending the generated trigger signals to the control layer, which then instructs the control layer on how to handle subsequent interactions.

[0094] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0095] Step S201: Obtain historical dialogue data from the dialogue memory storage area, wherein the dialogue memory storage area is used to maintain the historical information of multiple rounds of dialogue between the target object within a preset time range.

[0096] In this embodiment, the dialogue memory storage area is used to maintain the multi-turn dialogue history information of the target object within a preset time range. It is understood that this application proposes a multi-level memory architecture, and the dialogue memory storage area is a component of this architecture, namely, dialogue memory, used to store multi-turn dialogue history information related to a specific scenario or environment. It is also understood that maintaining the multi-turn dialogue history information of the target object within the preset time range involves not only storing the information but also a series of management and processing tasks related to it, such as dynamic updating, efficient retrieval, and in-depth understanding and utilization of the information. These are not limited here.

[0097] Multi-turn dialogue history can be understood as the history of a user's interactions with a multimodal interaction system on a certain hardware device (such as smart glasses or smartwatches), or a record of interactions conducted by the user in a specific location (such as home, office, or coffee shop). There are no restrictions on this.

[0098] The preset time range can be a pre-defined time window, usually a recent period of time (such as the past few hours, day, or week), thereby maintaining the timeliness and relevance of scene memory.

[0099] In this embodiment, retrieving historical dialogue data from the dialogue memory storage area can be understood as the smart device accessing the dialogue memory storage area, determining which part of the dialogue memory needs to be called based on the current interaction scenario, and then retrieving the relevant historical dialogue data from it. This historical dialogue data will be used in subsequent decision-making processes to generate responses that are more tailored to the user's scenario and needs. For example, the control layer in a multi-agent interaction framework can retrieve historical dialogue data from the dialogue scenario memory storage area; this is not limited to this specific scenario.

[0100] It can be seen that by using dialogue memory, multimodal interaction systems can provide more contextualized services based on the user's current environment and recent interaction history. At the same time, by utilizing historical dialogue data, multimodal interaction systems can establish the contextual chain of user dialogues, ensuring that responses to users are not only accurate but also coherent, avoiding problems such as "fragmentation" or "contextual asynchrony".

[0101] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0102] Step S202: In response to receiving authorization from the target object, the historical dialogue data is preprocessed to obtain a processing result, wherein the processing result is used to describe the target object's preference content;

[0103] Step S203: Store the processing result in the long-term memory storage area. The long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through active recording or passive recording. Active recording records the target object's preference content according to the target object's instructions. Passive recording uses an information extraction model to extract the target object's preference content from the target object's dialogue history information.

[0104] It is understood that the multi-level memory architecture proposed in this application embodiment may also include a long-term memory storage area, i.e., long-term memory. The long-term memory storage area is used to store important information such as user preference content for a long time, and to perform a series of management and processing on user preference content. Furthermore, the long-term memory storage area is used to maintain the preference content of the target object from the perspective of the target object, through active recording or passive recording.

[0105] The active recording method records the target user's preferences based on the target user's instructions. This can be understood as the user directly informing the multimodal interaction system of their preferences that they want to be remembered, and the multimodal interaction system records them accordingly. For example, a user can give the instruction "Remember this document I'm looking at," and the multimodal interaction system will then actively store the document in long-term memory.

[0106] The passive recording method uses an information extraction model to extract the target object's preferences from the target object's dialogue history. This can be understood as a multimodal interaction system using an information extraction model to analyze the user's dialogue history in multiple situations. The information extraction model automatically identifies and learns the user's preferences and stores them in long-term memory without explicit user instruction.

[0107] Authorization of the target user can be understood as the user granting the multimodal interaction system permission to process their personal information (including historical conversation data), which usually follows privacy policies and data protection regulations.

[0108] Preprocessing can include cleaning, format conversion, and feature extraction of historical dialogue data to make it suitable for subsequent processing; this is not limited. Preprocessing can also include text segmentation, speech transcription, image recognition, and using natural language processing techniques and other data analysis methods to understand the meaning of user expressions. The preprocessed results are used to describe information about user preferences for subsequent personalized interactions; these results may include, for example, favorite music genres and frequently visited restaurants; this is not limited.

[0109] In this embodiment, the acquired historical dialogue data can be refined and filtered before being stored in long-term memory. For example, the control layer in the multi-agent interaction framework first performs a series of preprocessing operations on the historical dialogue data to extract user preferences and obtain processing results describing those preferences. Then, the control layer stores the processing results in long-term memory, ensuring the long-term preservation of user preference information. This allows the multimodal interaction system to continuously respond according to user preferences even during interactions across devices and time periods.

[0110] As can be seen, the long-term memory storage area allows user preferences to be shared across different scenarios and devices, ensuring that the personalized services provided by the multimodal interaction system remain consistent in any situation. Furthermore, as users spend more time using the system, the preferences stored in long-term memory are continuously enriched and updated, thereby enabling continuous optimization and evolution of the service.

[0111] In an optional embodiment, in step S23, session analysis is performed on the multimodal session content to obtain a sequence of actions to be executed, including the following method steps:

[0112] Step S231: Identify the target object's interaction intent from the multimodal conversation content using a semantic rule engine;

[0113] Step S232: In a known scenario, control the fine-tuned action planning model to generate a sequence of actions to be executed based on the interaction intent; or, in an unknown scenario, control the zero-sample action planning model to generate a sequence of actions to be executed based on the interaction intent.

[0114] It is understandable that the multi-agent interaction framework of a multimodal interaction system can also include a planning layer. The planning layer is used to receive and analyze the multimodal conversation content sent by the control layer, break down the user's instructions into a list of actions to be executed and attach corresponding parameters.

[0115] In this embodiment, when performing conversation analysis on multimodal conversation content to obtain a sequence of actions to be executed, a semantic rule engine can be used to identify the interaction intent of the target object from the multimodal conversation content. The semantic rule engine can be a tool based on predefined grammatical rules and semantic models, used to parse input multimodal data and understand user intent. The semantic rule engine can process and recognize different forms of information such as text, speech, and images, and transform user input into executable commands or requests through pattern matching and semantic analysis. For example, the semantic rule engine can be a traditional semantic understanding engine based on Lightweight Grammar Formalism (LGF) grammars, matching user intent through exhaustive methods. The semantic rule engine can also parse user instructions using Speech Act Theory to identify the specific speech act type behind them, thereby responding to user intent more accurately; however, this is not limited to specific examples.

[0116] Interaction intent can be understood as the action goals, needs, or requests expressed by users through multimodal conversation content.

[0117] The fine-tuned action planning model can be understood as a large action planning model that has undergone fine-tuning (FT). It's used to collect data for user command scenarios and then fine-tunes the large model to obtain a list of actions. Fine-tuning is a common step in machine learning, especially deep learning. It involves further training a pre-trained model using a small amount of data specific to a particular task, allowing the model to better adapt to the characteristics of that task or specific dataset. In essence, it means that under a known scenario, the pre-trained action planning model is specifically fine-tuned to better adapt to the interpretation and response planning of user commands in a particular scenario, thus obtaining the fine-tuned action planning model.

[0118] Zero-shot action planning models can generate action items using a strong base model and text prompts. This can be understood as follows: in unknown or new scenarios, zero-shot action planning models can generate a reasonable sequence of actions to be executed based on the user's interaction intent, even with limited or no specific scenario data.

[0119] In this application, identifying the target object's interaction intent from multimodal conversation content using a semantic rule engine can be understood as the planning layer in the multi-agent interaction framework using the semantic rule engine to deeply analyze the multimodal conversation content and identify the user's specific interaction intent. For example, if a user says, "I want to know tomorrow's weather," and simultaneously displays a screenshot of a weather forecast, the semantic rule engine will combine the speech recognition and image recognition results to determine that the user wants to know about tomorrow's weather conditions; this is not a limitation. Therefore, multimodal information can be accurately parsed, ensuring that the multimodal interaction system accurately understands the user's intent and avoids misunderstandings or improper handling.

[0120] Subsequently, the planning layer can control the fine-tuned action planning model to generate a sequence of actions to be executed based on the interaction intent in a known scenario, or control the zero-shot action planning model to generate a sequence of actions to be executed based on the interaction intent in an unknown scenario. In other words, the planning layer will choose to use either the fine-tuned action planning model or the zero-shot action planning model to generate the sequence of actions to be executed based on the identified interaction intent.

[0121] Specifically, in known scenarios, such as when users frequently inquire about the weather at home, the planning layer uses an action planning model fine-tuned based on the data from that scenario to generate a response, thereby enabling a more accurate and rapid response.

[0122] In unknown scenarios, such as when a user makes a new request that the multimodal interaction system has never encountered before, the planning layer will use a zero-shot action planning model. This zero-shot action planning model can handle new scenarios more flexibly and generate action sequences based on a wider range of data and knowledge.

[0123] It can be seen that by dynamically selecting the fine-tuned action planning model and the zero-shot action planning model, the multimodal interaction system can generate reasonable responses in both known and unknown scenarios, thereby improving scenario adaptability and generalization ability.

[0124] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0125] Step S204: Based on the interaction intent, determine whether to use a multimodal search method to recall the memory fragment of the interaction intent from the long-term memory storage area. The long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through active recording or passive recording. Active recording records the target object's preference content according to the target object's instructions. Passive recording uses an information extraction model to extract the target object's preference content from the target object's dialogue history information. The memory fragment is used to generate personalized responses in conjunction with the action sequence to be executed.

[0126] Understandably, the planning layer also possesses tool invocation and memory activation discrimination functions. For example, the planning layer can use a text discrimination model to determine whether the current user command needs to enable search enhancement and whether the current user command needs to be associated with the user's past memories. If the determination is "needed," a multimodal search can be used to search for relevant content in the long-term memory storage area and send the relevant content to the downstream execution agent, which can then generate a response that matches the user's historical preferences.

[0127] In this embodiment, the multimodal search method can be understood as a method that combines multiple information types such as text, voice, images, and video for information retrieval, aiming to understand user needs more comprehensively and accurately and provide personalized services.

[0128] Memory fragments can be understood as historical records or preference information related to user interaction intentions stored in the long-term memory storage area. Memory fragments are used to generate personalized responses in conjunction with the sequence of actions to be performed, that is, they can be used as a reference when generating personalized responses.

[0129] Personalized responses can be understood as responses or suggestions that are highly matched to the user's specific needs, generated based on the user's preferences and historical behavior patterns.

[0130] In this application, determining whether to retrieve memory fragments of the interaction intent from long-term memory storage using a multimodal search method based on the interaction intent can be understood as the planning layer using a decision mechanism to decide whether to retrieve relevant information from long-term memory storage based on the user's expressed interaction intent. If the decision result is "needed," a multimodal search method will be used to find memory fragments related to the user's current request. The found memory fragments will be combined with the generated sequence of actions to be executed, thereby generating a response that is more tailored to the user's personality.

[0131] For example, if a user once said "I love hot pot", this content will be stored in the long-term memory storage area. A few days later, when the user says "What are some good places to eat nearby", the system will use a multimodal search method to recall the memory fragment of the interaction intent from the long-term memory storage area. The multimodal interaction system will then prioritize recommending hot pot-related items and highlight them in the reply.

[0132] It can be seen that by determining whether to recall memory fragments from long-term memory storage and combining this with enhanced multimodal search, the multimodal interaction system can provide more personalized and context-relevant responses, i.e., highly personalized services. This not only improves the user experience but also demonstrates the intelligence and flexibility of the multimodal interaction system in understanding user needs. Furthermore, the execution result is guided and influenced by the personalized response, meaning that the execution result can be generated based on user preferences and historical information, ensuring the targetedness and effectiveness of the action.

[0133] In an optional embodiment, in step S24, the sequence of actions to be executed is assigned to multiple agents for execution to obtain the execution result, including the following method steps:

[0134] Step S241: Distribute the action sequence to be executed to modally matched agents among multiple agents for independent execution, and / or distribute the action sequence to cross-modal agents among multiple agents for cooperative execution, and obtain the execution result.

[0135] It is understandable that the multi-agent interaction framework of a multimodal interaction system can also include an execution layer. For example, the execution layer can consist of a router, a large number of agents with various modalities, and dialogue functions. The router receives the action list output by the planning layer, i.e., the sequence of actions to be executed, and distributes the sequence of actions to different agents for execution. It is important to note that after an agent completes its execution, it returns the execution result to the planning layer. The planning layer then determines whether all the sequences of actions to be executed have been completed. After all the sequences of actions to be executed have been completed, all execution results are streamed back to the user.

[0136] In this embodiment, when the sequence of actions to be executed is assigned to multiple agents for execution and the execution result is obtained, the sequence of actions to be executed can be assigned to agents with matching modalities for independent execution. This can be understood as the execution layer assigning each instruction in the sequence of actions to be executed to an agent specializing in processing data of a particular modality. For example, text instructions might be sent to a text processing agent, and voice instructions might be sent to a voice processing agent; this is not limited here. Therefore, agents with matching modalities can quickly and accurately process data corresponding to their own modality, reducing latency and errors in cross-modal processing, while providing higher quality execution results.

[0137] Understandably, after independent execution, the execution results from different agents will be integrated. For example, based on task requirements and user preferences, a comprehensive judgment will be made to select or merge the most relevant results to form a unified feedback strategy.

[0138] And / or, the sequence of actions to be executed is distributed to cross-modal agents among multiple agents for collaborative execution to obtain the execution result. This can be understood as follows: for tasks requiring the combination of multiple modal data, the execution layer assigns the sequence of actions to agents capable of cross-modal collaboration. These agents share information and work together to complete complex tasks. Thus, cross-modal agents can combine multiple information types to provide more comprehensive and accurate understanding and execution capabilities. For example, when processing a request involving both voice and images, they can provide better suggestions through comprehensive analysis. Simultaneously, through cooperation among multiple agents, complex tasks involving multiple modalities can be executed, improving the functionality and user satisfaction of multimodal interaction systems.

[0139] For example, based on the modal characteristics involved in a series of action instructions to be executed, such as whether they include speech recognition, image analysis, or text understanding, these instructions can be intelligently assigned to appropriate cross-modal agents. Each agent then independently and efficiently completes the task within its specialized modal domain. During this process, agents can share necessary information, such as keywords, contextual understanding, or user preferences, through internal system communication mechanisms, thereby enhancing each other's processing capabilities and decision-making quality. Once all agents have completed their assigned tasks, the execution results from these different agents are integrated to form a unified, coherent, and insightful feedback, which is ultimately presented to the user, ensuring that the interaction process is both rich and diverse, and closely aligned with the user's specific needs and context.

[0140] For example, when both independent and collaborative execution exist, action instructions involving different modalities can be precisely assigned. For instance, voice instructions are assigned to a voice agent, and image tasks to a vision agent, ensuring that each agent can independently and in parallel process its specialized modality task. Simultaneously, through coordination, cross-modal agents can share necessary information, conduct collaborative execution, and ultimately integrate the results of all independent and collaborative executions to provide the user with a comprehensive and coherent response.

[0141] As can be seen, this application considers that user-side input may include various information types such as text, audio, and visual information. Therefore, a cross-modal multi-agent scheme is designed at the execution layer, enabling agents of different modalities to autonomously complete tasks they are good at, or to interact and cooperate to complete tasks. Through the combined use of independent agents and cross-modal agents, the multimodal interaction system can efficiently and accurately process user requests, providing a personalized and high-quality service experience. It also enables the multimodal interaction system to handle a wider range of tasks, demonstrating the flexibility and scalability of the multimodal interaction system's intelligence. Furthermore, through the specialization and collaboration of agents, the multimodal interaction system can allocate resources more effectively, reduce redundant calculations, and improve the speed and efficiency of task processing.

[0142] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0143] Step S205: Store the execution result in the scene memory storage area, whereby the scene memory storage area is used to temporarily store the execution results of different actions of multiple intelligent agents.

[0144] It is understood that the multi-level memory architecture proposed in this application embodiment may also include a scene memory storage area, i.e., scene memory. The scene memory storage area is used to temporarily store the execution results of intelligent agents related to the current interaction scene, that is, to temporarily store the execution results of different actions of multiple intelligent agents. It can be understood as a form of short-term memory used to support ongoing dialogues or tasks.

[0145] In this embodiment, storing the execution result in the scene memory storage area can be understood as the execution layer storing the execution result in the scene memory storage area. For example, if a user requests through smart glasses, "Teach me how to make this dish," the multimodal interaction system may invoke a voice assistant agent for dialogue understanding and guidance, while simultaneously invoking a visual recognition agent to analyze the recipe image. After the agents complete their respective tasks, the execution results are sent back to the scene memory storage area, which may include "the ingredients needed for this dish," "voice guidance on the preparation steps," and "key information from the recipe image," etc., without limitation.

[0146] Therefore, the scene memory storage area can provide temporary data caching for the current interaction scene, ensuring that all relevant action results are recorded during task execution or within the interaction loop for later use or reference. This helps maintain the contextual coherence of the dialogue or task and avoids information loss. At the same time, the information stored in the scene memory can be accessed and integrated by subsequent agents or planning layers, helping the multimodal interaction system make more comprehensive and accurate decisions.

[0147] In an optional embodiment, step S25 involves feeding back the execution result to the target object, including the following method steps:

[0148] Step S251: Determine whether all actions in the action sequence to be executed have been completed;

[0149] In step S252, in response to the completion of all multiple actions, the execution results are fed back to the target object using a streaming method.

[0150] In this embodiment, streaming transmission is a data transmission method that can continuously send data and is suitable for scenarios that require real-time interaction and feedback.

[0151] In this application, when feeding back the execution results to the target object, it can first be determined whether all actions in the sequence of actions to be executed have been completed. In response to the completion of all actions, the execution results are fed back to the target object using a streaming method. This can be understood as the multimodal interaction system confirming whether all agents have completed their assigned tasks; that is, the multimodal interaction system checks whether each action in the sequence of actions to be executed has been correctly executed and completed by the agent to ensure that all execution results are collected. If the multimodal interaction system confirms that all agents have completed their tasks, it will use a streaming method to feed back the execution results to the target object in real time to ensure the timeliness and continuity of information transmission.

[0152] Therefore, streaming transmission ensures timely feedback of execution results, reducing user waiting time. This is especially beneficial for scenarios requiring immediate responses, such as online shopping inquiries and navigation guidance, greatly enhancing the smoothness and satisfaction of the user experience. In other words, this application's multimodal interaction system not only ensures that all user-requested operations are fully executed, but also presents the execution results to the user in the smoothest and most timely manner.

[0153] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0154] Step S206: In response to meeting preset conditions, the number of agents and / or the multimodal processing capabilities of the multiple agents are expanded.

[0155] In this embodiment of the application, in response to the satisfaction of preset conditions, the number of agents and / or the multimodal processing capabilities of multiple agents are expanded. This can be understood as the multimodal interaction system continuously evaluating the current operating status and user needs. Once the preset conditions are detected, the multimodal interaction will automatically expand the number of agents and / or the multimodal processing capabilities of multiple agents.

[0156] Among them, meeting the preset conditions may include increased complexity of user requests (e.g., input content includes multiple types of information), high current load pressure of the multimodal interaction system, and high resource utilization (e.g., the utilization rate of computing power and storage space is close to the preset upper limit), etc., which are not restricted here.

[0157] Through the aforementioned extension mechanism, it is ensured that the multimodal interaction system can dynamically adjust resources according to actual needs, so as to handle increasingly complex or high-concurrency interaction tasks more efficiently.

[0158] For example, in a smart cockpit scenario, if the multimodal interaction system detects multiple passengers simultaneously issuing different interaction requests (such as adjusting seats, checking traffic conditions, or playing music), and these requests involve data from multiple modalities, the system will decide whether to increase the number of agents or enhance the multimodal processing capabilities of a single agent based on preset load management and resource optimization strategies. Thus, by increasing the number of agents, the multimodal interaction system can handle more user requests simultaneously, supporting stable operation in high-concurrency scenarios and avoiding response delays or task failures due to insufficient processing power. Simultaneously, by expanding multimodal processing capabilities, the system can more flexibly respond to complex or changing interaction scenarios, supporting more diverse user needs and interaction methods, thereby improving the service quality and user experience of the multimodal interaction system.

[0159] It can be seen that the multi-agent interaction framework of this application is forward-looking, taking into account the scalability of the full-modal agent, enabling the full-modal agent to adapt to more complex and broader multimodal interaction needs when necessary by adding new functions, improving processing power or optimizing algorithms.

[0160] Figure 3 This is a schematic diagram illustrating the operation of a multimodal, multi-agent interaction framework according to an embodiment of this application, such as... Figure 3 As shown, the external environment includes the user, the user's environment information, and dynamic knowledge (such as real-time weather, traffic conditions, etc.).

[0161] Multi-agent interaction frameworks can be used with different hardware devices (e.g. Figure 3The system utilizes intelligent cockpits, smart wearables, mobile phones, and smart screens as carriers to conduct real-time, multi-modal interactions with users. Specifically, the multi-agent interaction framework can include four layers: perception layer, control layer, planning layer, and execution layer.

[0162] The perception layer, serving as the direct window for interaction between the entire multimodal interaction system and the user, receives multimodal information input from the device in real time, performs perception and understanding of each modality, and generates trigger signals based on the understanding results, sending these signals along with the modal content to the control layer. For example, the perception layer may employ sound activity detection technology to identify sound events (such as when a user begins or stops speaking), but it can also use speech recognition technology, video perception technology, etc., for perception and understanding; this is not a limitation. It is understood that the multimodal interaction system can use different technologies for perception and understanding to generate trigger signals as needed.

[0163] The control layer receives trigger signals from the perception layer and manages the history of short-term conversations. Combining user device information and historical records, it organizes the information into multimodal content and sends it to the planning layer. On the other hand, with authorization, the control layer also refines, filters, and stores the user's conversation history information in the long-term memory module.

[0164] The planning layer receives and analyzes the multimodal conversation content input from the control layer, breaking down user commands into sequences of actions to be executed and attaching corresponding parameters. Specifically, the planning layer includes the following modules: a semantic rule engine, a fine-tuned action planning model, a zero-shot action planning model, and tool invocation and memory activation discrimination. For detailed descriptions, please refer to the aforementioned sections; they will not be repeated here.

[0165] The execution layer consists of a router and a large number of agents with various modalities and dialogue functions. The router receives the sequence of actions to be executed from the planning layer and distributes the sequence of actions to different agents (e.g., Figure 3 The system executes actions using various intelligent agents, including voice assistants, visual question-answering agents, bartending / chess teaching agents, voice translation agents, and interactive podcast agents. After each agent completes its execution, it returns the result to the planning layer. The planning layer then determines whether all actions have been completed and, if so, streams all execution results back to the user.

[0166] Based on the aforementioned multi-agent interaction framework, this application achieves the following three important features through the addition of some key components:

[0167] The first type is a multi-level memory storage system.

[0168] This application designs a multi-level memory architecture including scene memory, dialogue memory, and long-term memory to record user logs and preference information. Among them, scene memory is used for temporary storage of single action execution, dialogue memory is used to maintain the user's multi-turn dialogue history in the short term, and long-term memory records the user's preference content in an active or passive manner, based on the user.

[0169] The second type is personalized interaction.

[0170] With the support of a multi-level memory architecture, multimodal interaction systems can become increasingly intelligent during the interaction process. Specifically, this application uses a tool call and memory activation discrimination module at the planning layer to determine whether the user's current command requires memory enhancement. If the determination is "yes", a multimodal search is used to search for relevant content in the long-term memory storage area and send the content to the downstream execution agent. The agent can then generate a response that matches the user's historical preferences.

[0171] The third type is cross-modal multi-agent systems.

[0172] Considering that user-side input may include multimodal information such as text, audio, and visual data, this application designs a cross-modal multi-agent scheme at the execution layer. This allows different modal agents to autonomously complete tasks they are good at, or to interact and cooperate to complete tasks. Furthermore, the multi-agent interaction framework of this application also considers the scalability of the full-modal agent, enabling it to adapt to more complex and broader multimodal interaction needs by adding new features, improving processing capabilities, or optimizing algorithms when necessary.

[0173] For example, the above steps can be implemented using a full-modal understanding and generative model combined with a long-term memory multimodal search enhancement framework, without limitation here.

[0174] It is easy to understand that the beneficial effects of the intelligent agent communication and interaction method provided in this application include the following points.

[0175] Beneficial Effect (1): This application proposes a multi-level memory structure that generates a user-personalized multimodal knowledge base by accepting the user's active memory instructions and passively mining and analyzing the user's dialogue history (under authorization). When the user interacts with the multimodal interaction system, the planning layer will make a decision on whether to activate long-term memory based on the current user instructions and multimodal information, and recall the corresponding memory fragments to generate a personalized response.

[0176] Beneficial effect (2): This application realizes multimodal interaction by splitting multiple modules, giving full play to the potential of various single-modal better models and intelligent agents, and ensuring intervention and business scalability.

[0177] Beneficial effect (3): This application designs a cross-modal multi-agent structure, which supports the interaction of multiple agents while preserving the better interaction effect of different modalities as much as possible. For example, in the voice assistant scenario, when the user says "teach me to play chess", the video chess teaching agent will be activated to interact with the user.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0179] Furthermore, it should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0181] According to embodiments of this application, the following are also provided: Figure 4 This illustrates a method for intelligent agent communication and interaction. Figure 4 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application, such as... Figure 4 As shown, the method includes:

[0182] Step S41: Generate a trigger signal based on the current input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object;

[0183] Step S42: Determine the multimodal merchant recommendation conversation content using the trigger signal and pre-stored historical merchant recommendation dialogue data;

[0184] Step S43: Perform conversation analysis on the multimodal merchant recommendation conversation content to obtain the merchant recommendation action sequence;

[0185] Step S44: Distribute the merchant recommendation action sequence to multiple agents for execution to obtain the merchant recommendation execution result;

[0186] Step S45: Feedback the merchant recommendation execution results to the target audience.

[0187] In this embodiment, a multimodal merchant recommendation request can be understood as a user expressing a need for merchant recommendations to the multimodal interaction system through multiple modalities (such as voice, text, and images). For example, saying "What are some good restaurants nearby?" via voice is not limited here.

[0188] Historical merchant recommendation dialogue data can be understood as the dialogue records and results of a user's past merchant recommendation requests stored in the long-term memory of a multimodal interaction system, including information such as user preferences, historical feedback, and choices.

[0189] The merchant recommendation action sequence can be understood as a series of action instructions generated by the multimodal interaction system based on the conversation analysis results. These instructions guide the subsequent agent to perform specific merchant recommendation tasks, such as filtering conditions, sorting rules, and presentation methods.

[0190] The results of merchant recommendation execution can be understood as the specific results such as recommendation lists, map markers, and voice responses obtained by multiple intelligent agents after performing merchant recommendation tasks according to the assigned action sequence.

[0191] For a detailed description, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0192] The intelligent agent communication and interaction method provided in this application embodiment can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0193] By employing the embodiments of this application, a trigger signal is generated based on the current input multimodal merchant recommendation request. The multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object. Then, the trigger signal and pre-stored historical merchant recommendation dialogue data are used to determine the multimodal merchant recommendation session content. Next, session analysis is performed on the multimodal merchant recommendation session content to obtain a merchant recommendation action sequence. This sequence is then distributed to multiple agents for execution, yielding merchant recommendation execution results. Finally, the execution results are fed back to the target object. This achieves efficient cross-modal multi-agent collaboration, providing a more natural, efficient, and personalized multimodal interaction for various scenarios. This solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0194] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0195] According to embodiments of this application, the following are also provided: Figure 5 This illustrates a method for intelligent agent communication and interaction. Figure 5 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application, such as... Figure 5 As shown, the method includes:

[0196] Step S51: Obtain the agent communication interaction request through the first application programming interface. The request data carried in the agent communication interaction request includes: multimodal information, which is determined according to the multimodal interaction mode of the target object.

[0197] Step S52: Return the agent communication interaction response through the second application programming interface. The response data carried in the agent communication interaction response includes: the execution result, which is obtained according to any of the above agent communication interaction methods.

[0198] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0199] In this embodiment of the application, the Application Programming Interface (API) is a software interface that allows different software applications to interact. The API defines how applications communicate, exchange data, and call services.

[0200] A communication interaction request from an intelligent agent can be understood as a request sent from another intelligent agent or application, requesting the multimodal interaction system to perform a specific multimodal interaction task or operation. For example, a request might be sent from the voice assistant intelligent agent of smart glasses, asking the multimodal interaction system to identify the image currently being viewed by the user and provide relevant information.

[0201] Intelligent agent communication interaction response can be understood as the result returned to the requester (intelligent agent or application) through the API interface after the multimodal interaction system has completed the requested task, which includes specific feedback or information on the execution of the task.

[0202] For a detailed description, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0203] The intelligent agent communication and interaction method provided in this application embodiment can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0204] By employing the embodiments of this application, an agent communication interaction request is obtained through a first application programming interface (API). The request data carried in the agent communication interaction request includes multimodal information, which is determined based on the multimodal interaction method of the target object. Then, an agent communication interaction response is returned through a second API. The response data carried in the agent communication interaction response includes an execution result, which is obtained according to any of the above agent communication interaction methods. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios. This solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0205] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0206] According to embodiments of this application, the following are also provided: Figure 6 This illustrates a method for intelligent agent communication and interaction. Figure 6 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application, such as... Figure 6 As shown, the method includes:

[0207] Step S61: Obtain the currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, and the multimodal information is determined according to the multimodal interaction mode of the target object;

[0208] Step S62: In response to the agent communication interaction dialogue request, return the agent communication interaction dialogue reply, wherein the information carried in the agent communication interaction dialogue reply includes: execution result, the execution result is obtained according to any of the above agent communication interaction methods;

[0209] Step S63: Display the execution results in the graphical user interface.

[0210] In this embodiment, the intelligent agent communication interaction dialogue request can be understood as a request from another intelligent agent or user, requiring the multimodal interaction system to respond to a multimodal dialogue task, such as answering questions, providing suggestions, or performing an operation. The request data contains multimodal information, that is, it may contain various types of data such as text, voice, and images.

[0211] Intelligent agent communication and interactive dialogue responses can be understood as feedback or answers generated by a multimodal interaction system in response to received multimodal requests. The information carried in the response includes the execution result, that is, the specific output generated by the multimodal interaction system after performing the task according to the request, such as a recommendation list, question answers, operation confirmation, etc.

[0212] A graphical user interface (GUI) is a user interface that uses graphical elements (such as buttons, sliders, menus, and windows) to display information and receive user input. It is widely used in smart devices and applications, providing an intuitive and convenient user operating environment.

[0213] For a detailed description, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0214] The intelligent agent communication and interaction method provided in this application embodiment can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0215] By employing the embodiments of this application, the system acquires the currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, which is determined according to the multimodal interaction method of the target object. Then, in response to the intelligent agent communication interaction dialogue request, an intelligent agent communication interaction dialogue reply is returned, wherein the information carried in the intelligent agent communication interaction dialogue reply includes: execution result, which is obtained according to any of the above-mentioned intelligent agent communication interaction methods. Finally, the execution result is displayed in the graphical user interface, thereby achieving the goal of efficient cross-modal multi-agent collaboration. This realizes the technical effect of providing more natural, more efficient, and more personalized multimodal interaction for the needs of multiple scenarios, and solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration between multiple agents and cannot meet diverse scenario needs and personalized requirements.

[0216] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0217] According to embodiments of this application, the following are also provided: Figure 7 This illustrates a method for intelligent agent communication and interaction. Figure 7 This is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this application, such as... Figure 7 As shown, the method includes:

[0218] Step S71: In response to the input command applied to the operation interface, display multimodal information on the operation interface;

[0219] Step S72: In response to the processing command applied to the operation interface, display the execution result on the operation interface; wherein the execution result is obtained according to any of the above-mentioned agent communication interaction methods.

[0220] In this embodiment of the application, the operation interface can be understood as the interface through which a user interacts with a smart device or application, and may be a touch screen, a voice control interface or other forms of user input interface.

[0221] Input commands can be understood as commands issued by the user through the user interface, used to provide operation or query requests to a multimodal interaction system. In multimodal interaction scenarios, input commands may include various forms of data such as voice, text, and images.

[0222] Processing instructions can be understood as instructions given by users or intelligent agents to the multimodal interaction system to perform specific operations or tasks through the user interface, such as searching for information, playing music, or setting reminders.

[0223] For a detailed description, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0224] The intelligent agent communication and interaction method provided in this application embodiment can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0225] By adopting the embodiments of this application, multimodal information is displayed on the operation interface in response to input commands applied to the operation interface, and then the execution result is displayed on the operation interface in response to processing commands applied to the operation interface. The execution result is obtained according to any of the above-mentioned agent communication and interaction methods, thereby achieving the goal of efficient cross-modal multi-agent collaboration. This achieves the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios, and solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0226] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0227] According to embodiments of this application, the following are also provided: Figure 8 The diagram shows an intelligent agent communication and interaction system. Figure 8 This is a schematic diagram of the structure of an intelligent agent communication and interaction system according to an embodiment of this application, such as... Figure 8 As shown, the system includes:

[0228] The perception layer is used to generate trigger signals based on the multimodal information of the current input, wherein the multimodal information is determined according to the multimodal interaction mode of the target object;

[0229] The control layer is used to determine the content of a multimodal conversation using trigger signals and pre-stored historical dialogue data;

[0230] The planning layer is used to perform session analysis on the multimodal session content to obtain the sequence of actions to be executed;

[0231] The execution layer is used to distribute the sequence of actions to be executed to multiple agents for execution and to obtain the execution results.

[0232] The planning layer is also used to feed back the execution results to the target object.

[0233] In this embodiment, the intelligent agent communication and interaction system can execute the above-described intelligent agent communication and interaction method, as detailed in the description of the foregoing embodiments, which will not be repeated here.

[0234] The intelligent agent communication and interaction system provided in this application embodiment can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0235] By adopting the embodiments of this application, the intelligent agent communication and interaction system achieves the goal of efficient collaboration among cross-modal multi-agents, thereby realizing the technical effect of providing more natural, efficient and personalized multi-modal interaction for the needs of multiple scenarios. This solves the technical problem in related technologies that multi-modal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0236] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0237] According to embodiments of this application, the following are also provided: Figure 9 The diagram shows an intelligent agent communication and interaction system. Figure 9 This is a schematic diagram of the structure of an intelligent agent communication and interaction system according to an embodiment of this application, such as... Figure 9 As shown, the system includes:

[0238] The client is used to send multimodal information, which is determined based on the multimodal interaction method of the target object;

[0239] The server connects to the client and is used to generate trigger signals based on multimodal information, determine the multimodal conversation content using the trigger signals and pre-stored historical dialogue data, perform conversation analysis on the multimodal conversation content to obtain the sequence of actions to be executed, and assign the sequence of actions to be executed to multiple agents for execution to obtain the execution result.

[0240] The client is also used to output the execution results.

[0241] In this embodiment, the intelligent agent communication and interaction system can execute the above-described intelligent agent communication and interaction method, as detailed in the description of the foregoing embodiments, which will not be repeated here.

[0242] The intelligent agent communication and interaction system provided in this application embodiment can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0243] By adopting the embodiments of this application, the intelligent agent communication and interaction system achieves the goal of efficient collaboration among cross-modal multi-agents, thereby realizing the technical effect of providing more natural, efficient and personalized multi-modal interaction for the needs of multiple scenarios. This solves the technical problem in related technologies that multi-modal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0244] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0245] According to an embodiment of this application, an apparatus embodiment for implementing the above-described intelligent agent communication interaction method is also provided. Figure 10 This is a schematic diagram of the structure of an intelligent agent communication and interaction device according to an embodiment of this application, such as... Figure 10 As shown, the device includes:

[0246] The first generation module 101 generates a trigger signal based on the currently input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object;

[0247] The first determining module 102 uses a trigger signal and pre-stored historical merchant recommendation dialogue data to determine the content of the multimodal merchant recommendation conversation;

[0248] The first analysis module 103 performs conversation analysis on the multimodal merchant recommendation conversation content to obtain the merchant recommendation action sequence;

[0249] The first execution module 104 distributes the merchant recommendation action sequence to multiple intelligent agents for execution, and obtains the merchant recommendation execution result;

[0250] The first feedback module 105 feeds back the results of the merchant's recommendation to the target audience.

[0251] Optionally, the first generation module 101 is further configured to: perform perception and understanding on the currently input multimodal information to obtain an understanding result, wherein the understanding result is used to perceive the intention or action of the target object through the multimodal information; and generate a trigger signal based on the understanding result.

[0252] Optionally, the intelligent agent communication interaction method further includes: obtaining historical dialogue data from a dialogue memory storage area, wherein the dialogue memory storage area is used to maintain the historical information of multiple rounds of dialogue of the target object within a preset time range.

[0253] Optionally, the intelligent agent communication interaction method further includes: in response to receiving authorization from the target object, preprocessing historical dialogue data to obtain processing results, wherein the processing results are used to describe the target object's preference content; storing the processing results in a long-term memory storage area, wherein the long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through an active recording method or a passive recording method. The active recording method records the target object's preference content according to the target object's instructions, while the passive recording method uses an information extraction model to extract the target object's preference content from the target object's dialogue history information.

[0254] Optionally, the first analysis module 103 is further configured to: identify the interaction intent of the target object from the multimodal conversation content through a semantic rule engine; in a known scenario, control the fine-tuned action planning model to generate a sequence of actions to be executed based on the interaction intent; or, in an unknown scenario, control the zero-sample action planning model to generate a sequence of actions to be executed based on the interaction intent.

[0255] Optionally, the intelligent agent communication interaction method further includes: determining whether to use a multimodal search method to recall memory fragments of the interaction intent from the long-term memory storage area based on the interaction intent, wherein the long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through an active recording method or a passive recording method. The active recording method records the target object's preference content according to the target object's instructions, while the passive recording method uses an information extraction model to extract the target object's preference content from the target object's dialogue history information. The memory fragments are used to generate personalized responses in conjunction with the sequence of actions to be executed.

[0256] Optionally, the first execution module 104 is further configured to: assign the action sequence to be executed to modally matched agents among multiple agents for independent execution, and / or assign the action sequence to be executed to cross-modal agents among multiple agents for cooperative execution, thereby obtaining the execution result.

[0257] Optionally, the intelligent agent communication and interaction method further includes: storing the execution result in a scene memory storage area, wherein the scene memory storage area is used to temporarily store the execution results of different actions of multiple intelligent agents.

[0258] Optionally, the first feedback module 105 is further configured to: determine whether all actions in the action sequence to be executed have been completed; and, in response to the completion of all actions, feed back the execution results to the target object using a streaming transmission method.

[0259] Optionally, the agent communication interaction method further includes: in response to satisfying preset conditions, expanding the number of agents and / or the multimodal processing capabilities of multiple agents.

[0260] By employing the embodiments of this application, a trigger signal is generated based on the multimodal information of the current input. The multimodal information is determined according to the multimodal interaction mode of the target object. Then, the multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data. Next, the multimodal conversation content is analyzed to obtain a sequence of actions to be executed. The sequence of actions to be executed is then assigned to multiple agents for execution to obtain the execution result. Finally, the execution result is fed back to the target object. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios. This solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0261] It should be noted that the first generation module 101, the first determination module 102, the first analysis module 103, the first execution module 104, and the first feedback module 105 mentioned above correspond to steps S21 to S25 in the embodiments. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiments.

[0262] According to an embodiment of this application, another apparatus embodiment for implementing the above-described intelligent agent communication interaction method is also provided. Figure 11 This is a schematic diagram of another intelligent agent communication and interaction device according to an embodiment of this application, such as... Figure 11 As shown, the device includes:

[0263] The second generation module 111 generates a trigger signal based on the currently input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object;

[0264] The second determining module 112 uses the trigger signal and pre-stored historical merchant recommendation dialogue data to determine the content of the multimodal merchant recommendation conversation;

[0265] The second analysis module 113 performs conversation analysis on the multimodal merchant recommendation conversation content to obtain the merchant recommendation action sequence;

[0266] The second execution module 114 distributes the merchant recommendation action sequence to multiple intelligent agents for execution, and obtains the merchant recommendation execution result;

[0267] The second feedback module 115 feeds back the merchant recommendation execution results to the target object;

[0268] By employing the embodiments of this application, a trigger signal is generated based on the current input multimodal merchant recommendation request. The multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object. Then, the trigger signal and pre-stored historical merchant recommendation dialogue data are used to determine the multimodal merchant recommendation session content. Next, session analysis is performed on the multimodal merchant recommendation session content to obtain a merchant recommendation action sequence. This sequence is then distributed to multiple agents for execution, yielding merchant recommendation execution results. Finally, the execution results are fed back to the target object. This achieves efficient cross-modal multi-agent collaboration, providing a more natural, efficient, and personalized multimodal interaction for various scenarios. This solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0269] It should be noted that the second generation module 111, the second determination module 112, the second analysis module 113, the second execution module 114, and the second feedback module 115 mentioned above correspond to steps S41 to S45 in the embodiments. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiments.

[0270] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described intelligent agent communication and interaction method is also provided. Figure 12 This is a schematic diagram of the structure of another intelligent agent communication and interaction device according to an embodiment of this application, such as... Figure 12 As shown, the device includes:

[0271] The first acquisition module 121 acquires an agent communication interaction request through a first application programming interface. The request data carried in the agent communication interaction request includes multimodal information, which is determined based on the multimodal interaction method of the target object.

[0272] The first return module 122 returns an agent communication interaction response through the second application programming interface. The response data carried in the agent communication interaction response includes: the execution result, which is obtained according to any of the above agent communication interaction methods.

[0273] By employing the embodiments of this application, an agent communication interaction request is obtained through a first application programming interface (API). The request data carried in the agent communication interaction request includes multimodal information, which is determined based on the multimodal interaction method of the target object. Then, an agent communication interaction response is returned through a second API. The response data carried in the agent communication interaction response includes an execution result, which is obtained according to any of the above agent communication interaction methods. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios. This solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0274] It should be noted that the first acquisition module 121 and the first return module 122 mentioned above correspond to steps S51 to S52 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0275] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described intelligent agent communication and interaction method is also provided. Figure 13 This is a schematic diagram of the structure of another intelligent agent communication and interaction device according to an embodiment of this application, such as... Figure 13 As shown, the device includes:

[0276] The second acquisition module 131 acquires the currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, and the multimodal information is determined according to the multimodal interaction mode of the target object;

[0277] The second return module 132, in response to the agent communication interaction dialogue request, returns an agent communication interaction dialogue response, wherein the information carried in the agent communication interaction dialogue response includes: the execution result, which is obtained according to any of the above agent communication interaction methods;

[0278] Display module 133 displays the execution results within the graphical user interface.

[0279] By employing the embodiments of this application, the system acquires the currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, which is determined according to the multimodal interaction method of the target object. Then, in response to the intelligent agent communication interaction dialogue request, an intelligent agent communication interaction dialogue reply is returned, wherein the information carried in the intelligent agent communication interaction dialogue reply includes: execution result, which is obtained according to any of the above-mentioned intelligent agent communication interaction methods. Finally, the execution result is displayed in the graphical user interface, thereby achieving the goal of efficient cross-modal multi-agent collaboration. This realizes the technical effect of providing more natural, more efficient, and more personalized multimodal interaction for the needs of multiple scenarios, and solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration between multiple agents and cannot meet diverse scenario needs and personalized requirements.

[0280] It should be noted that the second acquisition module 131, the second return module 132, and the display module 133 mentioned above correspond to steps S61 to S63 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0281] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described intelligent agent communication and interaction method is also provided. Figure 14 This is a schematic diagram of the structure of another intelligent agent communication and interaction device according to an embodiment of this application, such as... Figure 14 As shown, the device includes:

[0282] The first display module 141 responds to input commands applied to the operation interface and displays multimodal information on the operation interface;

[0283] The second display module 142 responds to the processing instructions applied to the operation interface and displays the execution result on the operation interface; wherein the execution result is obtained according to the intelligent agent communication interaction method of any one of claims 1 to 10.

[0284] By adopting the embodiments of this application, multimodal information is displayed on the operation interface in response to input commands applied to the operation interface, and then the execution result is displayed on the operation interface in response to processing commands applied to the operation interface. The execution result is obtained according to any of the above-mentioned agent communication and interaction methods, thereby achieving the goal of efficient cross-modal multi-agent collaboration. This achieves the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios, and solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0285] It should be noted that the first display module 141 and the second display module 142 mentioned above correspond to steps S71 to S72 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0286] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in the embodiments, but are not limited to the schemes provided in the embodiments.

[0287] Embodiments of this application may provide a computing device. Figure 15 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 15 As shown, the computing device A may include one or more ( Figure 15 (Only one is shown) processor 1502, memory 1504, memory controller, and peripheral interface, wherein the peripheral interface may connect to radio frequency module, audio module, display screen, etc., which are not limited here.

[0288] The aforementioned computing device A can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0289] Specifically, computing device A can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, computing device A can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, computing device A also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, computing device A can also create applications based on models, providing API calling capabilities, allowing models to be called into created applications through API interfaces, and providing application management tools for application management and monitoring.

[0290] Furthermore, the computing device A can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0291] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0292] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0293] Those skilled in the art will understand that, Figure 15 The structure shown is for illustrative purposes only; computing device A can also be a smartphone, tablet, PDA, mobile internet device (MID), PAD, or other terminal device. Figure 15This does not limit the structure of the aforementioned computing device. For example, computing device A may also include components that are more advanced than those described above. Figure 15 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same as the Figure 15 The different configurations shown.

[0294] Embodiments of this application may provide an electronic device. Figure 16 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 16 As shown, the electronic device may include: an input / output device 162; a memory 164; and a processor 166, wherein the processor 166 is connected to the input / output device 162 and the memory 164 via a bus 168.

[0295] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0296] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0297] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0298] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the intelligent agent communication and interaction method provided in the above embodiments.

[0299] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0300] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described intelligent agent communication and interaction methods.

[0301] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0302] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0303] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0304] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0305] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0306] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for intelligent agent communication and interaction, characterized in that, include: A trigger signal is generated based on the current input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; The multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data; The multimodal conversation content is analyzed to obtain a sequence of actions to be executed; The sequence of actions to be executed is assigned to multiple agents for execution, and the execution results are obtained. The execution result is then fed back to the target object.

2. The intelligent agent communication and interaction method according to claim 1, characterized in that, Generating the trigger signal based on the currently input multimodal information includes: The multimodal information currently input is perceptually understood to obtain an understanding result, wherein the understanding result is used to perceive the intention or action of the target object through the multimodal information; The trigger signal is generated based on the understanding results.

3. The intelligent agent communication and interaction method according to claim 1 or 2, characterized in that, The intelligent agent communication and interaction method further includes: The historical dialogue data is obtained from the dialogue memory storage area, wherein the dialogue memory storage area is used to maintain the historical dialogue information of the target object in multiple rounds within a preset time range.

4. The intelligent agent communication and interaction method according to claim 1 or 2, characterized in that, The intelligent agent communication and interaction method further includes: In response to receiving authorization from the target object, the historical dialogue data is preprocessed to obtain a processing result, wherein the processing result is used to describe the target object's preference content; The processing result is stored in a long-term memory storage area, wherein the long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through an active recording method or a passive recording method. The active recording method records the target object's preference content according to the target object's instructions, and the passive recording method uses an information extraction model to extract the target object's preference content from the target object's dialogue history information.

5. The intelligent agent communication and interaction method according to claim 1 or 2, characterized in that, The multimodal session content is analyzed to obtain the sequence of actions to be executed, which includes: The semantic rule engine identifies the interaction intent of the target object from the multimodal session content; In a known scenario, the finely tuned action planning model generates the sequence of actions to be executed based on the interaction intent; or, in an unknown scenario, the zero-sample action planning model generates the sequence of actions to be executed based on the interaction intent.

6. The intelligent agent communication and interaction method according to claim 5, characterized in that, The intelligent agent communication and interaction method further includes: Based on the interaction intent, it is determined whether to use a multimodal search method to recall the memory fragment of the interaction intent from the long-term memory storage area. The long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through an active recording method or a passive recording method. The active recording method records the target object's preference content according to the target object's instructions. The passive recording method uses an information extraction model to extract the target object's preference content from the target object's dialogue history information. The memory fragment is used to generate a personalized response in conjunction with the action sequence to be executed.

7. The intelligent agent communication and interaction method according to claim 1 or 2, characterized in that, The sequence of actions to be executed is assigned to the multiple agents for execution, and the execution results include: The sequence of actions to be executed is assigned to modality-matching agents among the multiple agents for independent execution, and / or the sequence of actions to be executed is assigned to cross-modality agents among the multiple agents for cooperative execution, thereby obtaining the execution result.

8. The intelligent agent communication and interaction method according to claim 7, characterized in that, The intelligent agent communication and interaction method further includes: The execution results are stored in the scene memory storage area, which is used to temporarily store the execution results of different actions of the multiple intelligent agents.

9. The intelligent agent communication and interaction method according to claim 1 or 2, characterized in that, Feeding the execution result back to the target object includes: Determine whether all actions in the sequence of actions to be executed have been completed; In response to the completion of all the aforementioned actions, the execution results are fed back to the target object using a streaming transmission method.

10. The intelligent agent communication and interaction method according to claim 1 or 2, characterized in that, The intelligent agent communication and interaction method further includes: In response to the fulfillment of preset conditions, the number of intelligent agents and / or the multimodal processing capabilities of the multiple intelligent agents are expanded.

11. A method for intelligent agent communication and interaction, characterized in that, include: A trigger signal is generated based on the current input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object; The trigger signal and pre-stored historical merchant recommendation dialogue data are used to determine the content of the multimodal merchant recommendation session; The multimodal merchant recommendation conversation content is analyzed to obtain the merchant recommendation action sequence; The merchant recommendation action sequence is distributed to multiple intelligent agents for execution, and the merchant recommendation execution result is obtained. The results of the merchant recommendation will be fed back to the target object.

12. A method for intelligent agent communication and interaction, characterized in that, include: The intelligent agent communication interaction request is obtained through the first application programming interface, wherein the request data carried in the intelligent agent communication interaction request includes: multimodal information, which is determined according to the multimodal interaction mode of the target object; The agent communication interaction response is returned through the second application programming interface, wherein the response data carried in the agent communication interaction response includes: the execution result, which is obtained according to the agent communication interaction method according to any one of claims 1 to 10.

13. A method for intelligent agent communication and interaction, characterized in that, include: Obtain the currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, wherein the multimodal information is determined based on the multimodal interaction mode of the target object; In response to the agent communication interaction dialogue request, an agent communication interaction dialogue response is returned, wherein the information carried in the agent communication interaction dialogue response includes: an execution result, the execution result being obtained according to the agent communication interaction method according to any one of claims 1 to 10; The execution results are displayed within the graphical user interface.

14. A method for intelligent agent communication and interaction, characterized in that, include: In response to input commands applied to the user interface, multimodal information is displayed on the user interface. In response to processing instructions applied to the operation interface, the execution result is displayed on the operation interface; The execution result is obtained according to the intelligent agent communication and interaction method described in any one of claims 1 to 10.

15. An intelligent agent communication and interaction system, characterized in that, include: The perception layer is used to generate trigger signals based on the multimodal information of the current input, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; The control layer is used to determine the multimodal session content using the trigger signal and pre-stored historical dialogue data; The planning layer is used to perform session analysis on the multimodal session content to obtain a sequence of actions to be executed; The execution layer is used to distribute the sequence of actions to be executed to multiple agents for execution, and obtain the execution results. The planning layer is also used to feed back the execution results to the target object.

16. An intelligent agent communication and interaction system, characterized in that, include: A client is used to send multimodal information, wherein the multimodal information is determined based on the multimodal interaction method of the target object; The server, connected to the client, is used to generate a trigger signal based on the multimodal information, determine the multimodal conversation content using the trigger signal and pre-stored historical dialogue data, perform conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed, and assign the sequence of actions to be executed to multiple agents for execution to obtain execution results. The client is also used to output the execution result.

17. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program executes the intelligent agent communication and interaction method according to any one of claims 1 to 14 when it runs.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device containing the computer-readable storage medium to perform the intelligent agent communication interaction method according to any one of claims 1 to 14.

19. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the intelligent agent communication and interaction method according to any one of claims 1 to 14.

Images