Agent-based task planning methods and intelligent systems

Abstract

This application discloses a task planning method and intelligent system based on intelligent agents, belonging to the field of artificial intelligence. Through the technical solution provided in the embodiments of this application, the main intelligent agent performs task planning on the target task based on the task description information of the target task, the first multimodal information from multimodal sensors, and the task reference information obtained through a long-term memory, obtaining first task planning information. This process incorporates the task reference information, making the obtained first task planning information more accurate. The main intelligent agent sends the first task planning information to multiple sub-intelligent agents. Each sub-intelligent agent executes corresponding actions based on the state information obtained from a short-term memory and the first task planning information. Because the sub-intelligent agents incorporate state information when executing actions, the actions executed by multiple sub-intelligent agents are less likely to conflict, resulting in better completion of the target task.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a task planning method and intelligent system based on intelligent agents. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent agent systems are being used more and more widely in fields such as industrial automation and robotics.

[0003] Traditional task planning methods often employ a centralized decision-making architecture, where a single agent generates task sequences based on preset rules or environmental perception data.

[0004] However, in practical applications, it has been found that there are often problems such as insufficient information sharing and rigid behavior coordination mechanisms among the intelligent agents, resulting in poor task execution. Summary of the Invention

[0005] This application provides a task planning method and intelligent system based on intelligent agents, which can improve the performance of intelligent agents in performing tasks. The technical solution is as follows:

[0006] On the one hand, an agent-based task planning method is provided, applied to an intelligent system, the intelligent system including a main agent and multiple sub-agents, the method including:

[0007] The main intelligent agent performs task planning on the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory bank, and obtains the first task planning information.

[0008] The main intelligent agent sends the first task planning information to multiple sub-intelligent agents;

[0009] Each of the sub-agents retrieves state information from a short-term memory and performs corresponding actions based on the state information and the first task planning information to cooperate in completing the target task. The state information is used to represent the agent state of the multiple sub-agents.

[0010] In one possible implementation, the main intelligent agent performs task planning on the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained from the long-term memory, to obtain first task planning information, including:

[0011] The main intelligent agent encodes the task description information, the first multimodal information, and the task reference information using the multimodal large model to obtain the task description features of the task description information, the first multimodal features of the first multimodal information, and the task reference features of the task reference information.

[0012] The main intelligent agent fuses the task description features, the first multimodal features, and the task reference features through the multimodal large model to obtain the first multimodal fusion feature;

[0013] The main intelligent agent decodes the first multimodal fusion feature through the multimodal large model to obtain the first task planning information.

[0014] In this implementation, the main intelligent agent uses a multimodal large model to process the task description information, the first multimodal information, and the task reference information to obtain the first task planning information. The first task planning information is essentially a fusion of the original task description information, the collected first multimodal information, and the task reference information obtained through retrieval enhancement, and has a high degree of matching with the target task and the actual situation of the intelligent system.

[0015] In one possible implementation, the main agent encodes the task description information, the first multimodal information, and the task reference information using the multimodal large model to obtain task description features of the task description information, first multimodal features of the first multimodal information, and task reference features of the task reference information, including:

[0016] The main agent encodes the task description information and the task reference information using the text encoder of the multimodal large model based on an attention mechanism to obtain the task description features of the task description information and the task reference features of the task reference information. It also encodes the first multimodal information using the multimodal encoder of the multimodal large model based on an attention mechanism to obtain the first multimodal features of the first multimodal information.

[0017] In this implementation, using a multimodal large model to encode task description information, task reference information, and first multimodal information based on an attention mechanism can achieve more expressive encoding, thereby obtaining more expressive features.

[0018] In one possible implementation, the main intelligent agent performs task planning for the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory. Before obtaining the first task planning information, the method further includes:

[0019] The main intelligent agent queries the long-term memory for related task description information based on the task description information.

[0020] The main intelligent agent fills the task description information and the associated task description information into the prompt text template to obtain the search-enhanced prompt text;

[0021] The main agent inputs the search enhancement prompt text into the search enhancement model, and processes the search enhancement prompt text through the search enhancement model to obtain the task reference information.

[0022] In this implementation, associated task information is retrieved from the long-term memory, achieving efficient acquisition of such information. Task description information and associated task description information are then populated into a text template to obtain enhanced search suggestion text. This enhanced search suggestion text is processed using a search enhancement model to obtain task reference information, thus fully utilizing the information in the long-term memory and resulting in highly accurate task reference information.

[0023] In one possible implementation, each of the sub-agents obtains state information from a short-term memory and performs corresponding actions based on the state information and the first task planning information, including:

[0024] For any one of the plurality of sub-agents, the sub-agent obtains state information from the short-term memory and obtains the sub-planning information corresponding to the sub-agent from the first task planning information;

[0025] The sub-agent generates action execution information based on the state information and the sub-planning information;

[0026] The sub-agent performs the action indicated by the action execution information.

[0027] In this implementation, the sub-agents retrieve state information from short-term memory and corresponding sub-planning information from the first task planning information. Based on the state information and sub-planning information, action execution information is generated. Executing the actions indicated by the action execution information completes the corresponding sub-tasks. When multiple sub-agents complete their respective sub-tasks, the target task is thus accomplished.

[0028] In one possible implementation, the sub-agent generates action execution information based on the state information and the sub-planning information, including:

[0029] The sub-agent inputs the state information and the sub-planning information into the action execution information generation model, and encodes the state information and the sub-planning information through the action execution information generation model to obtain action execution features;

[0030] The sub-agent decodes the action execution features through the action execution information generation model to obtain the action execution information.

[0031] In this implementation, the sub-agent uses an execution information generation model to encode state information and sub-planning information into action execution features, and then decodes the action execution features through the action execution information generation model to obtain action execution information. Thus, a more accurate action execution information is obtained through the encoding and decoding process.

[0032] In one possible implementation, the method further includes:

[0033] The sub-agent synchronizes the agent's state to the short-term memory in real time.

[0034] In this implementation, the agent states stored in the short-term memory can be kept up-to-date, making it easier for each sub-agent to know the agent states of other sub-agents and facilitating cooperation among multiple sub-agents.

[0035] In one possible implementation, the method further includes:

[0036] In response to a fault occurring when the intelligent system is performing the target task, the main intelligent agent retrieves fault reference information from the fault memory based on the fault information of the fault.

[0037] The main intelligent agent re-plans the target task based on the task description information of the target task, the second multimodal information collected by the multimodal sensor, and the fault reference information, to obtain the second task planning information.

[0038] The master agent sends the second task planning information to the multiple sub-agents.

[0039] In this implementation, when a fault occurs in the intelligent system, the target task can be re-planned using fault reference information, task description information, and second multimodal information in the fault memory bank, thereby obtaining second task planning information. The second task planning information can be regarded as task planning information to eliminate the fault. After the second task planning information is sent to multiple sub-intelligent agents, the multiple sub-intelligent agents can use the second task planning information to repair the fault and continue to execute the target task, making the robustness of the intelligent system stronger.

[0040] In one possible implementation, the method further includes:

[0041] In response to a failure occurring when the intelligent system is performing the target task, the main intelligent agent determines the failure level, which is used to indicate the severity of the failure;

[0042] Based on the fault level, the master agent sends a control command to at least one of the plurality of sub-agents, the control command being used to control the sub-agent to stop executing the corresponding action.

[0043] In this implementation, if a failure occurs while performing the target task, the failure level is determined. The failure level is then used to send control commands to multiple sub-agents, causing the sub-agents receiving the control commands to cease performing the corresponding actions, thus preventing the impact of the failure from escalating.

[0044] On the one hand, an intelligent system is provided, which includes a main intelligent agent and multiple sub-intelligent agents;

[0045] The main intelligent agent is used to perform task planning for the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory bank, to obtain the first task planning information.

[0046] The main intelligent agent is also used to send the first task planning information to multiple sub-intelligent agents;

[0047] Each of the sub-agents is used to obtain state information from a short-term memory bank and to perform corresponding actions based on the state information and the first task planning information in order to cooperate in completing the target task. The state information is used to represent the agent state of the multiple sub-agents.

[0048] On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the agent-based task planning method.

[0049] On the one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of the intelligent system reads the program code from the computer-readable storage medium and executes the program code, causing the intelligent system to perform the above-described agent-based task planning method.

[0050] The technical solution provided in this application embodiment allows the main intelligent agent to perform task planning based on the task description information of the target task, the first multimodal information from the multimodal sensor, and the task reference information obtained from the long-term memory, thus obtaining first task planning information. This process incorporates the task reference information, resulting in higher accuracy of the obtained first task planning information. The main intelligent agent sends the first task planning information to multiple sub-intelligent agents. Each sub-intelligent agent executes corresponding actions based on the state information obtained from the short-term memory and the first task planning information. Because the sub-intelligent agents incorporate state information when executing actions, the actions executed by multiple sub-intelligent agents are less likely to conflict, leading to better completion of the target task. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the implementation environment of an agent-based task planning method provided in an embodiment of this application;

[0053] Figure 2 This is a flowchart of a task planning method based on an intelligent agent provided in an embodiment of this application;

[0054] Figure 3 This is a flowchart of another agent-based task planning method provided in an embodiment of this application;

[0055] Figure 4 This is a flowchart illustrating a method for determining task planning information provided in an embodiment of this application;

[0056] Figure 5 This is a flowchart of a memory classification method provided in an embodiment of this application;

[0057] Figure 6 This is a flowchart of a task planning method based on an intelligent agent provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0059] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0060] Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve better results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.

[0061] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0062] Intelligent System: An intelligent system is a comprehensive system based on artificial intelligence (AI) technology, capable of autonomously or semi-autonomously completing complex tasks and adapting to dynamically changing environments by perceiving the environment, processing information, learning, and reasoning. In the embodiments of this application, an intelligent system can refer to a robot, an intelligent driving system, or a system composed of multiple intelligent agents.

[0063] An agent is an intelligent entity capable of autonomously perceiving its environment, making decisions, and executing tasks, possessing multimodal data fusion and collaboration capabilities. In the embodiments of this application, the main agent serves as the decision-making core, integrating multi-source information such as text, images, and memory systems to achieve efficient decomposition and distribution of complex tasks; sub-agents perform precise operations based on real-time states and distributed memory.

[0064] Multi-agent interaction: The process by which multiple agents work together to complete tasks or solve complex problems in a shared environment through communication, cooperation or competition.

[0065] Distributed real-time data sharing system: This is a high-efficiency architecture that supports concurrent read and write operations across multiple nodes and real-time data synchronization, eliminating single-point bottlenecks through decentralized design. Its core employs a publish-subscribe model (such as Kafka / RabbitMQ) and an in-memory database (such as Redis) to achieve millisecond-level response times, combined with version control (vector clock) and conflict resolution algorithms to ensure data consistency.

[0066] Retrieval Augmented Generative (RAG) is a technology that combines information retrieval and text generation. It retrieves relevant document fragments from external knowledge bases and then inputs them into a generative model to generate accurate answers. Its core process is: Retrieve → Augment → Generate. It dynamically calls upon the latest data, avoiding the knowledge stagnation problem of traditional models, generates content based on retrieval evidence, significantly reduces illusions, and supports mixed retrieval of structured and unstructured data.

[0067] Long-term memory: Used to store data with long time sensitivity. In this embodiment, the long-term memory is used to store task reference information, which is used to assist in task planning and has a long time sensitivity.

[0068] Short-term memory: used to store short-term data. In this embodiment, the short-term memory is used to store the state information of the agent.

[0069] Semantic features: Features used to represent the semantics expressed by text. Different texts can correspond to the same semantic features; for example, the text "How's the weather today?" and the text "How's the weather today?" can correspond to the same semantic feature. Intelligent systems can map characters in text to character vectors, and combine and operate on the character vectors according to the relationships between characters to obtain the semantic features of the text. For example, intelligent systems can use bidirectional encoder representations from transformers (BERT).

[0070] Normalization: Mapping sequences of values ​​with different ranges to the interval (0, 1) to facilitate data processing. In some cases, normalized values ​​can be directly expressed as probabilities.

[0071] Embedded coding, mathematically speaking, represents a correspondence, that is, mapping data in space X to space Y using a function F. Here, F is an injective function, and the mapping result preserves the structure. An injective function means that the mapped data uniquely corresponds to the original data, and preserving the structure means that the size relationship of the original data is the same as that of the mapped data. For example, if there are data X1 and X2 before mapping, after mapping we get Y1 corresponding to X1 and Y2 corresponding to X2. If the original data X1 > X2, then correspondingly, the mapped data Y1 > Y2. For words, this means mapping words to another space to facilitate subsequent machine learning and processing.

[0072] Attention weights represent the importance of a piece of data during training or prediction. Importance indicates the magnitude of the influence of input data on output data. Data with high importance corresponds to higher attention weights, while data with low importance corresponds to lower attention weights. The importance of data varies in different scenarios, and training the model to assign attention weights is essentially the process of determining data importance.

[0073] Figure 1 This is a schematic diagram illustrating the implementation environment of an agent-based task planning method provided in an embodiment of this application. See also... Figure 1 The implementation environment may include a main intelligent agent 101 and multiple sub-intelligent agents 102. The main intelligent agent 101 is used for overall task planning, and the sub-intelligent agents 102 are used to execute specific task actions, thereby cooperating to complete the corresponding tasks.

[0074] For example, in a scenario involving the control of a single robot, the robot comprises multiple motion components, such as the robot's arms, hands, legs, feet, neck, and body. The main agent 101 acts as the robot's global controller, while the sub-agents 102 act as local controllers, controlling the robot's motion components. Each motion component is controlled by one sub-agent 102. When the robot's objective is to overcome obstacles, the main agent 101 performs overall task planning, breaking down the objective into sub-tasks for each sub-agent 102. These sub-tasks are then distributed to their respective sub-agents, which execute their assigned tasks. Each sub-agent 102 then controls its controlled robot motion components to collaboratively complete the obstacle-overcoming objective.

[0075] For example, in the context of intelligent driving systems, these systems include visual information acquisition units, speed control units, steering control units, and lighting control units. The main intelligent agent 101 acts as the global controller of the intelligent driving system, while the sub-intelligent agents 102 act as local controllers, controlling the speed control unit, steering control unit, and lighting control unit. Each sub-intelligent agent 102 controls one unit. Through the cooperation of the main intelligent agent 101 and multiple sub-intelligent agents 102, corresponding intelligent driving tasks can be executed.

[0076] For example, in a scenario where multiple robots work together to complete a target task, the main intelligent agent 101 acts as the global controller for the multiple robots, while the sub-intelligent agent 102 controls a single robot. This scenario is commonly seen in logistics situations where multiple robots are used to complete the sorting of goods. Through the cooperation of multiple robots, efficient goods sorting can be achieved.

[0077] After introducing the implementation environment of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below. See [link to relevant documentation]. Figure 2 The method is applied to an intelligent system, which includes a main intelligent agent and multiple sub-intelligent agents. The method includes the following steps.

[0078] 201. Based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory, the main intelligent agent performs task planning for the target task and obtains the first task planning information.

[0079] The target task is the task that the intelligent system needs to perform. For example, in a scenario controlled by a single robot, the target task might be to grasp an item; in an autonomous driving scenario, the target task might be to change lanes and overtake; and in a multi-robot system scenario, the target task might be to sort multiple packages in a warehouse. The target task is set by technicians according to requirements. The master intelligent agent is the core of task planning for the target task. The task planning process can break down the target task into multiple sub-tasks, and the master intelligent agent distributes these sub-tasks to multiple sub-intelligent agents for execution, thereby completing the target task through the cooperation of multiple sub-intelligent agents. Multimodal sensors are used to collect multimodal information, including images and point clouds. The first multimodal information is collected by the multimodal sensors when performing the target task. The first multimodal information reflects the environmental conditions; for example, in a scenario controlled by a single robot, the first multimodal information reflects the environmental conditions around the robot; and in an autonomous driving scenario, the first multimodal information reflects the environmental conditions around the vehicle. The long-term memory stores task reference information for different tasks. This task reference information can be considered as prior information for performing the corresponding task. Combining the task reference information with the long-term memory enables better task planning for the target task. Furthermore, the task reference information stored in the long-term memory can be updated by technicians according to actual conditions to keep it up-to-date. The first task planning information is obtained by combining the first multimodal information and the task reference information to plan the target task. This first task planning information can be used to instruct the sub-agents to perform their tasks.

[0080] 202. The main agent sends the first task planning information to multiple sub-agents.

[0081] In this process, after the first task planning information is sent to multiple sub-agents, the multiple sub-agents can perform subsequent control based on the first task planning information.

[0082] 203. Each sub-agent retrieves state information from the short-term memory and performs corresponding actions based on the state information and the first task planning information to cooperate in completing the target task. The state information is used to represent the agent state of multiple sub-agents.

[0083] Specifically, the short-term memory stores the agent states of multiple sub-agents. This allows the agent states of other sub-agents to be known before a single sub-agent executes an action, thereby avoiding conflicts between actions performed by different sub-agents and improving the effectiveness of collaboration among multiple sub-agents.

[0084] The technical solution provided in this application embodiment allows the main intelligent agent to perform task planning based on the task description information of the target task, the first multimodal information from the multimodal sensor, and the task reference information obtained from the long-term memory, thus obtaining first task planning information. This process incorporates the task reference information, resulting in higher accuracy of the obtained first task planning information. The main intelligent agent sends the first task planning information to multiple sub-intelligent agents. Each sub-intelligent agent executes corresponding actions based on the state information obtained from the short-term memory and the first task planning information. Because the sub-intelligent agents incorporate state information when executing actions, the actions executed by multiple sub-intelligent agents are less likely to conflict, leading to better completion of the target task.

[0085] It should be noted that steps 201-203 above are a brief introduction to the technical solutions provided in the embodiments of this application. The technical solutions provided in the embodiments of this application will be described in more detail below with some examples. See [link to relevant documentation]. Figure 3 Taking an intelligent system as the executing entity as an example, the method includes the following steps.

[0086] 301. The main intelligent agent obtains the task reference information of the target task through the long-term memory.

[0087] In one possible implementation, the main agent queries related task description information in long-term memory based on the task description information of the target task. The main agent then populates the prompt text template with the task description information and related task description information to obtain retrieval-enhanced prompt text. The main agent inputs the retrieval-enhanced prompt text into a retrieval enhancement model, which processes the text to obtain task reference information. This task reference information is used to assist in task planning.

[0088] The task description information describes the requirements of the target task. This information can be in natural language, and technicians can configure or adjust it to enable the intelligent system to understand the target task more accurately. Associated task description information is related to the target task; it includes descriptions of other tasks related to it. This information assists in task planning, leading to better planning results. In some embodiments, the associated task description information also includes performance baselines for multiple sub-agents, a decision logic graph of the main agent, and an instruction semantic tree. The performance baseline describes the functions and capabilities of the sub-agents, specifying what actions they can perform and the extent of those actions. The decision logic graph represents the relationships between different tasks. The instruction semantic tree includes task planning information for multiple tasks and the corresponding task execution status, including success or failure. The process of retrieving associated task description information from the long-term memory is the retrieval process in retrieval enhancement. The number of associated task descriptions can be one or more. The prompt text template is used to generate prompt text. It contains two slots: one for task description information and the other for related task description information. The prompt text template is set by technical personnel according to requirements or obtained through prompt word engineering. The search-enhanced prompt text is text used for search enhancement processing. The search enhancement process includes two steps: retrieval to find relevant content, and enhancement to enrich the found relevant content for subsequent processing. The search enhancement model is a model with search enhancement capabilities, and can employ the RAG model from related technologies. In some embodiments, the task description information is obtained by text recognition of audio collected by a multimodal sensor. The audio collected by the multimodal sensor can be user command audio, that is, the user instructs the system to perform the target task via audio.

[0089] In this implementation, the main agent queries the long-term memory using the task description information of the target task to obtain associated task description information. The main agent then populates the prompt text template with the task description information and associated task description information to obtain the retrieval-enhanced prompt text. The retrieval-enhanced text is processed using a retrieval enhancement model to obtain the task reference information of the target task, which can be used for subsequent task planning.

[0090] For example, the main agent extracts features from the task description information of the target task, obtaining task description features. The main agent matches these task description features with the task description features of multiple candidate task descriptions stored in long-term memory to determine the associated task description information from among the candidate information. The feature similarity between the task description features of the associated task description information and the task description features of the target task description information meets a preset similarity condition. The main agent fills the task description information and associated task description information into a prompt text template, obtaining the retrieval-enhanced prompt text. The main agent inputs the retrieval-enhanced prompt text into a retrieval enhancement model, which extracts features from the prompt text, obtaining the prompt text features. The main agent then uses the retrieval enhancement model to iteratively decode the prompt text features multiple times, obtaining task reference information.

[0091] Among them, the feature similarity meeting the preset similarity condition means that the feature similarity is greater than or equal to the similarity threshold, or the feature similarity is among the top N from high to low, where N is a positive integer. The similarity threshold (e.g., set to 0.9, with dynamic adjustment within the range of 0.1 to 1) and N are set by technical personnel according to their needs.

[0092] For example, the main agent encodes the task description information of the target task based on an attention mechanism, obtaining task description features. The main agent determines the feature similarity between these task description features and the task description features of multiple candidate task descriptions stored in long-term memory. The main agent identifies candidate task descriptions with feature similarity greater than or equal to a preset similarity threshold, or those ranking among the top N candidate task descriptions in terms of feature similarity from highest to lowest, as associated task descriptions. The main agent fills the task description information and associated task descriptions into a prompt text template, obtaining retrieval-enhanced prompt text. The main agent inputs the retrieval-enhanced prompt text into a retrieval enhancement model, which encodes the retrieval-enhanced prompt text based on an attention mechanism, obtaining prompt text features. The main agent then iteratively decodes the prompt text features through the retrieval enhancement model using an attention mechanism to obtain task reference information.

[0093] In this embodiment, the attention-based encoding is implemented through the encoder of the retrieval enhancement model, and the multi-round iterative decoding based on the attention mechanism is implemented through the decoder of the retrieval enhancement model. The structure of the encoder and decoder is not limited in this application. Compared with traditional information retrieval methods, retrieval enhancement has significant advantages: in terms of knowledge density, semantic-level matching is achieved through embedding spatial retrieval, replacing traditional keyword retrieval (such as BM25), effectively improving information utilization; in terms of multi-source information processing, it can integrate information from different structures and sources, enhancing the reasoning ability of complex tasks (such as multi-fault association analysis); however, the solution can be replaced with a less complex traditional matching mode, such as timestamp matching, keyword matching, or BM25, depending on the complexity of the task.

[0094] Additionally, if no relevant information is retrieved from the long-term memory, a null value can be returned to avoid the impact of error messages on task planning.

[0095] 302. Based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory, the main intelligent agent performs task planning for the target task and obtains the first task planning information.

[0096] A multimodal sensor is used to acquire multimodal information, including images and point clouds. In this embodiment, the multimodal sensor supports synchronous / asynchronous acquisition from heterogeneous signal sources. In some embodiments, the first multimodal information includes first visual information, which includes a first image and a first point cloud. The first image is acquired by the image acquisition unit of the multimodal sensor, and the first point cloud is acquired by the radar unit of the multimodal sensor.

[0097] In one possible implementation, the main agent encodes the task description information, the first multimodal information, and the task reference information using a multimodal large model to obtain task description features of the task description information, first multimodal features of the first multimodal information, and task reference features of the task reference information. The main agent then fuses the task description features, the first multimodal features, and the task reference features using the multimodal large model to obtain a first multimodal fused feature. Finally, the main agent decodes the first multimodal fused feature using the multimodal large model to obtain first task planning information.

[0098] Among them, the multimodal large model is a large language model with multimodal data processing capabilities. In the embodiments of this application, multimodal information includes visual information, which includes images and point clouds. That is, in the embodiments of this application, the multimodal large model can process text and visual information.

[0099] In this implementation, the main intelligent agent uses a multimodal large model to process the task description information, the first multimodal information, and the task reference information to obtain the first task planning information. The first task planning information is essentially a fusion of the original task description information, the collected first multimodal information, and the task reference information obtained through retrieval enhancement, and has a high degree of matching with the target task and the actual situation of the intelligent system.

[0100] To provide a clearer explanation of the above embodiments, the following description is divided into several parts.

[0101] In the first part, the main intelligent agent encodes the task description information, the first multimodal information, and the task reference information through a multimodal large model to obtain the task description features of the task description information, the first multimodal features of the first multimodal information, and the task reference features of the task reference information.

[0102] In the intelligent system provided in this application embodiment, the data input to the multimodal large model includes a regular input channel and an emergency input channel. The regular input channel is used to input the above-mentioned task description information, first multimodal information and task reference information into the multimodal large model when the task planning is performed normally. The emergency input channel is used to input relevant information into the multimodal large model when a fault occurs, so as to trigger the re-planning of the task.

[0103] In one possible implementation, the main agent encodes the task description information and task reference information using a text encoder of a multimodal large model based on an attention mechanism to obtain task description features of the task description information and task reference features of the task reference information. It also encodes the first multimodal information using a multimodal encoder of a multimodal large model based on an attention mechanism to obtain the first multimodal features of the first multimodal information.

[0104] Both the text encoder and the multimodal encoder are trained encoders. They map information to the same semantic space, facilitating subsequent feature processing. The multimodal encoder includes an image encoding unit and a point cloud encoding unit. The image encoding unit can be ViT (Vision Transformer), and the point cloud encoding unit can be PointNet, etc.

[0105] It should be noted that the above encoding process can be executed sequentially or synchronously. Furthermore, in addition to acquiring images and point clouds, multimodal sensors can also collect data from modalities such as tactile and olfactory information. Technicians can expand the hardware structure of multimodal sensors according to their needs, thereby enabling the acquisition of even more modal information.

[0106] In the second part, the main intelligent agent fuses the task description features, the first multimodal features, and the task reference features through a multimodal large model to obtain the first multimodal fused features.

[0107] The first multimodal feature includes image features and point cloud features.

[0108] In one possible implementation, the main agent uses a multimodal large model to fuse task description features, first multimodal features, and task reference features based on a cross-attention mechanism to obtain the first multimodal fused feature. Here, the query used in the cross-attention mechanism comes from one sequence, while the key and value come from another sequence. The task description features, first multimodal features, and task reference features can all be considered as a single sequence.

[0109] In this implementation, the cross-attention mechanism is used to fuse task description features, first multimodal features, and task reference features. The resulting first multimodal fused feature can fully model the dependencies between the fused task description features, first multimodal features, and task reference features, and has strong expressive power.

[0110] The above implementation method is illustrated below with two examples.

[0111] Example 1: The main agent uses a multimodal large model and a cross-attention mechanism to fuse task description features and first multimodal features to obtain the first fused feature. The main agent also uses a multimodal large model and a cross-attention mechanism to fuse the first fused feature and task reference features to obtain the first multimodal fused feature.

[0112] In the first fusion process described above, the task description feature serves as the query, and the first multimodal feature provides the key and value. In the second fusion process, the first fused feature serves as the query, and the task reference feature provides the key and value. Providing the key and value means multiplying the feature by the key transformation matrix and the value transformation matrix, respectively. Both the key transformation matrix and the value transformation matrix are linear transformation matrices, obtained through training.

[0113] Additionally, it should be noted that the above description is based on the example of fusing the task description feature and the first multimodal feature during the first fusion. In other possible implementations, any two other features can be fused during the first fusion, and the remaining feature can be fused during the second fusion. This application does not limit this.

[0114] Example 2: The main agent uses a multimodal large model and a cross-attention mechanism to fuse task description features and first multimodal features to obtain a first fused feature. The main agent uses the multimodal large model and a cross-attention mechanism to fuse task description features and task reference features to obtain a second fused feature. The main agent uses the multimodal large model and a cross-attention mechanism to fuse the first multimodal features and task reference features to obtain a third fused feature. The main agent concatenates the first, second, and third fused features to obtain the first multimodal fused feature.

[0115] The third part involves the main intelligent agent decoding the first multimodal fusion feature through the multimodal large model to obtain the first task planning information.

[0116] The first task planning information includes sub-planning information corresponding to multiple sub-agents. This sub-planning information indicates the sub-tasks assigned to the corresponding sub-agents, and the completion of multiple sub-tasks signifies the completion of the target task. The task planning process is essentially the process of breaking down the target task into sub-tasks corresponding to each sub-agent. The above-mentioned task planning for the target task is achieved through the encoding and decoding capabilities of a multimodal large model.

[0117] In one possible implementation, the main agent uses a multimodal large model to perform multiple rounds of iterative decoding on the first multimodal fusion features based on an attention mechanism to obtain the first task planning information.

[0118] The first task planning information includes action execution information of multiple sub-agents. The action execution information is used to indicate the actions that the sub-agents need to perform and the timing of the action execution. For example, the multiple action execution information in the first task planning information is stored in a certain order, and the order of the action execution information in the first task planning information can be used to indicate the timing of the action execution indicated by the action execution information.

[0119] In addition, after acquiring the first multimodal information, the main agent can also perform the following steps:

[0120] In one possible implementation, the main agent preprocesses the first multimodal information to obtain preprocessed first multimodal information, and then uses the preprocessed first multimodal information for task planning.

[0121] The first multimodal information includes first visual information, which includes a first image and a first point cloud. The first image can be processed using wavelet denoising, and the first point cloud can be preprocessed using voxel structuring. Of course, besides the above preprocessing methods, other preprocessing methods from related technologies can also be used, and this application embodiment does not limit them. The processes described in the above three parts can be... Figure 4 The process shown is used to illustrate this.

[0122] In this implementation, by preprocessing the acquired first multimodal information, errors and / or redundancies in the first multimodal information can be eliminated, thereby improving the effectiveness of subsequent task planning.

[0123] 303. The main intelligent agent sends the first task planning information to multiple sub-intelligent agents.

[0124] In this embodiment, after the first task planning information is sent to multiple sub-agents, the sub-agents can perform subsequent control based on the first task planning information. In this application embodiment, the intelligent system includes a main control channel and a collaborative channel. The main control channel is used for information transmission between the main agent and the sub-agents. For example, the main agent can send the first task planning information to the sub-agents through the main control channel, and the sub-agents can provide feedback on their agent status to the main agent through the main control channel. The collaborative channel is used for information sharing among the sub-agents; that is, in step 304 below, the sub-agents can share state information through the collaborative channel, which points to a short-term memory.

[0125] 304. Each sub-agent obtains state information from the short-term memory bank. The state information is used to represent the agent state of multiple sub-agents.

[0126] In this context, the agent state represents the current status of a sub-agent. For example, in a scenario where the technical solution provided in this application is used to control a single robot, multiple sub-agents are used to control movable components of the robot, such as its arms, hands, legs, feet, neck, and body. Taking the sub-agent controlling the arm as an example, its state reflects the arm's position and extension. Similarly, taking the sub-agent controlling the hand as an example, its state reflects the hand's position and finger extension. In a scenario where the technical solution provided in this application is used to control an intelligent driving system, multiple sub-agents are used to control a speed control unit, a steering control unit, and a lighting control unit. Taking the sub-agent controlling the speed control unit as an example, its state reflects the speed currently output by the speed control unit.

[0127] In a typical application scenario, multiple sub-agents are used to perform grasping tasks. That is, multiple sub-agents are used to control their respective grasping components to grasp the target object. When multiple sub-agents work in parallel, they can use the state information obtained from the short-term memory to determine the grasping state of other sub-agents in real time. In other words, multiple sub-agents can exchange grasping states and the pose information of the target object in real time through the short-term memory, thereby minimizing state conflicts between multiple sub-agents that could lead to the failure of the task of grasping the target object.

[0128] The state information in the short-term memory is updated in real time, meaning the states of multiple sub-agents are updated in real time. Each sub-agent can retrieve state information from the short-term memory, allowing it to understand the states of other sub-agents and facilitating action execution. In some embodiments, the short-term memory is also called a shared memory and can be implemented using Redis (Remote Dictionary Server).

[0129] 305. Each sub-agent executes corresponding actions based on its state information and the first task planning information to cooperate in completing the target task.

[0130] The short-term memory stores the agent states of multiple sub-agents. This allows a single sub-agent to know the agent states of other sub-agents before executing an action, thereby avoiding conflicts between actions performed by different sub-agents and improving the effectiveness of collaboration among multiple sub-agents.

[0131] In one possible implementation, for any one of the multiple sub-agents, the sub-agent retrieves state information from a short-term memory and sub-planning information corresponding to the sub-agent from the first task planning information. Based on the state information and sub-planning information, the sub-agent generates action execution information. The sub-agent then executes the action indicated by the action execution information.

[0132] In this context, the sub-planning information corresponding to each sub-agent is the planning information allocated to the sub-agent by the main agent when planning the target task. The sub-planning information corresponding to multiple sub-agents constitutes the first task planning information. The sub-task planning information instructs the sub-agents to execute their corresponding sub-tasks. When multiple sub-agents complete their respective sub-tasks, the target task is thus completed. For example, taking the target task as controlling a single robot to dance, multiple sub-tasks include controlling the robot's legs, arms, head, torso, feet, and hands, etc. These sub-tasks are obtained by breaking down the target task. Executing these sub-tasks synchronously completes the target task of "dancing."

[0133] In this implementation, the sub-agents retrieve state information from short-term memory and corresponding sub-planning information from the first task planning information. Based on the state information and sub-planning information, action execution information is generated. Executing the actions indicated by the action execution information completes the corresponding sub-tasks. When multiple sub-agents complete their respective sub-tasks, the target task is thus accomplished.

[0134] To provide a clearer explanation of the above embodiments, the following description is divided into several parts.

[0135] The first part involves the sub-agent obtaining state information from the short-term memory and obtaining the sub-planning information corresponding to the sub-agent from the first task planning information.

[0136] In one possible implementation, the sub-agent retrieves the latest state information from the short-term memory and uses its agent identifier to obtain the corresponding sub-planning information from the first task planning information. The agent identifier is used to uniquely identify the sub-agent; by using the agent identifier, the corresponding sub-agent can be located, thereby obtaining the corresponding sub-planning information.

[0137] The second part involves the sub-agent generating action execution information based on state information and sub-planning information.

[0138] In one possible implementation, the sub-agent inputs state information and sub-planning information into an action execution information generation model, encodes the state information and sub-planning information using the action execution information generation model to obtain action execution features. The sub-agent then decodes the action execution features using the action execution information generation model to obtain action execution information.

[0139] For example, the sub-agent inputs state information and sub-planning information into the action execution information generation model. The action execution information generation model then encodes the state information and sub-planning information using an attention mechanism to obtain action execution features. The sub-agent then uses the action execution information generation model to iteratively decode these action execution features multiple times using the attention mechanism to obtain the actual action execution information.

[0140] The action execution information generation model includes an encoder and a decoder, which are attention-based encoders and decoders, respectively. The encoder encodes state information and sub-planning information to obtain action execution features, and the decoder decodes these features to obtain action execution information. In other words, state information and sub-planning information can be considered as a raw sequence, and action execution information can be considered as a target sequence. The role of the action execution information generation model is to transform the raw sequence into the target sequence. Training the action execution information generation model means teaching the action execution information how to transform from the raw sequence to the target sequence.

[0141] Part Three: Sub-agents execute actions as instructed by the information.

[0142] In one possible implementation, the sub-agent executes action execution information to control the corresponding action component of the sub-agent to execute the corresponding action.

[0143] In some embodiments, based on the above implementation methods, the following steps can also be performed:

[0144] In one possible implementation, the sub-agent synchronizes the agent's state to the short-term memory bank in real time.

[0145] In some embodiments, after step 305, the following steps can also be performed.

[0146] 306. In response to a fault occurring when the intelligent system is performing a target task, the main intelligent agent retrieves fault reference information from the fault memory based on the fault information.

[0147] The fault memory database stores multiple fault reference information entries, which indicate the cause of the fault and the handling method. Utilizing this information enables better handling of faults. In some embodiments, the fault database stores fault logs from task execution, including logs of task failures and how technicians handled the faults.

[0148] In one possible implementation, in response to a failure occurring while the intelligent system is performing a target task, the master agent determines the failure level, which indicates the severity of the failure. Based on the failure level, the master agent sends a control command to at least one of a plurality of sub-agents, which instructs the sub-agent to stop performing the corresponding action.

[0149] In this implementation, if a failure occurs while performing the target task, the failure level is determined. The failure level is then used to send control commands to multiple sub-agents, causing the sub-agents receiving the control commands to cease performing the corresponding actions, thus preventing the impact of the failure from escalating.

[0150] To provide a clearer explanation of the above embodiments, the following description is divided into several parts.

[0151] Part 1: In response to a fault occurring when the intelligent system is performing a target task, the main intelligent agent determines the fault level.

[0152] In one possible implementation, in response to a failure occurring when the intelligent system is performing a target task, the main intelligent agent determines the fault level based on the fault code of the failure.

[0153] The fault code is a code generated when a fault occurs. The fault code indicates the type of fault. In this embodiment, the fault code is output by the fault detection system of the intelligent system. The fault code indicates the type of fault, and there is a correspondence between fault type and fault level. Therefore, the fault level can be determined using the fault code. The correspondence between fault type and fault level is set by technicians according to the actual situation, and this embodiment does not limit this.

[0154] In this implementation, when a fault occurs while the intelligent system is performing a target task, the fault code is used to determine the fault level, which is highly efficient.

[0155] For example, in response to a fault occurring while the intelligent system is performing a target task, the main intelligent agent retrieves the fault code output by the fault detection system. The main intelligent agent then uses the fault code to look up the corresponding fault type in a first relation table. Next, the main intelligent agent uses the fault type to look up the corresponding fault level in a second relation table. Alternatively, in response to a fault occurring while the intelligent system is performing a target task, the main intelligent agent retrieves the fault code output by the fault detection system. The main intelligent agent uses the fault code to look up the corresponding fault level in a third relation table.

[0156] The system consists of three relation tables: a first relation table storing multiple fault codes and their corresponding fault types, and a second relation table storing multiple fault types and their corresponding fault levels. Similarly, a third relation table storing multiple fault codes and their corresponding fault levels, and a fourth relation table storing multiple fault codes and their corresponding fault levels, and a fifth relation table storing multiple fault codes and their corresponding fault levels, and a sixth relation table storing multiple fault codes and their corresponding fault levels, and a seventh relation table storing multiple fault codes and their corresponding fault levels, and a eighth relation table storing multiple fault codes and their corresponding fault levels, and a ninth relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault codes and their corresponding fault levels, and a eleventh relation table storing multiple fault types ...

[0157] The following describes another implementation of the first part described above.

[0158] In one possible implementation, in response to a failure occurring when the intelligent system is performing a target task, the main intelligent agent determines the failure level based on the failure information.

[0159] The fault information describes the actual situation of the fault and can be in the form of natural language, which facilitates technical personnel to trace the fault back. In some embodiments, the fault information is output by the fault detection system. That is, after detecting a fault, the fault detection system generates and outputs the fault information, which the main intelligent agent can obtain.

[0160] In this implementation, the fault level is determined by using the fault information, and the fault level matches the actual situation of the fault more closely.

[0161] For example, in response to a fault occurring while the intelligent system is performing a target task, the main intelligent agent obtains the fault information output by the fault detection system. The main intelligent agent extracts features from the fault information to obtain fault description features. The main intelligent agent performs a fully connected operation and normalization on the fault description features to obtain the fault level.

[0162] For example, in response to a fault occurring while the intelligent system is performing a target task, the main agent acquires the fault information output by the fault detection system. The main agent encodes the fault information using an attention mechanism to obtain fault description features. The main agent then performs a fully connected and normalized process on these fault description features to obtain the fault level.

[0163] Here, "fully connected" refers to multiplying with a fully connected matrix, and "normalization" refers to substituting into a normalization function for processing. After normalization, the classification value corresponding to the fault description feature can be obtained. The main agent determines the target classification value interval to which the classification value belongs, and the fault level corresponding to the target classification value interval is determined as the fault level. That is, there are multiple fault levels, and each fault level corresponds to a classification value interval. The corresponding fault level can be determined by using the target classification value interval to which the classification value belongs.

[0164] Part Two: Based on the fault level, the master agent sends control commands to at least one of the multiple sub-agents.

[0165] In one possible implementation, when the fault level is greater than or equal to a first fault level threshold, the master agent sends control commands to multiple sub-agents. When the fault level is less than the first fault level threshold, the master agent, based on fault information, task reference information, and task description information of the target task, determines at least one target sub-agent from among the multiple agents. The master agent then sends control commands to the at least one target sub-agent.

[0166] In this application, a fault level greater than or equal to the first fault level threshold indicates a high fault level and a severe fault. In this case, control commands are sent to all sub-agents to stop their actions and prevent the impact of the severe fault from escalating. A fault level less than the first fault level threshold indicates a low fault severity. In this case, it is not necessary to send control commands to all sub-agents; instead, control commands are sent to the identified target sub-agent. The first fault level threshold is set by technicians based on actual conditions, and this application does not limit this setting.

[0167] In this implementation, when the fault level is high, control commands are sent to all sub-agents to instruct them to stop executing actions, thus preventing the impact of the severe fault from escalating. When the fault level is low, a target sub-agent is identified and a control command is sent to it to stop executing actions, thereby preventing the impact of the fault from escalating while maintaining the basic functions of the intelligent system.

[0168] To provide a clearer explanation of the above implementation methods, the method for determining at least one target sub-agent in the above implementation methods will be described below.

[0169] In one possible implementation, the main agent inputs fault information, task reference information, and target task description information into an agent recognition model. The agent recognition model then extracts features from these information to obtain sub-agent identification features. The main agent then performs a fully connected and normalized process on the sub-agent identification features using the agent recognition model to obtain at least one target sub-agent.

[0170] The task reference information carries the performance baselines of multiple sub-agents. The agent recognition model can identify the target sub-agent. The agent recognition model is trained based on multiple sample data and the corresponding labeled data for each sample data. The sample data includes sample fault information, sample task reference information, and sample task description information. The labeled data is used to indicate the corresponding target sub-agent.

[0171] For example, the main agent inputs fault information, task reference information, and the task description information of the target task into an agent recognition model. The model, based on an attention mechanism, encodes these information to obtain sub-agent identification features. The main agent then performs a fully connected and normalized process on these features using the agent recognition model, resulting in a probability set. This set contains multiple probabilities, each corresponding to a sub-agent, representing the likelihood of the sub-agent being identified as a target sub-agent. The main agent identifies the sub-agents corresponding to the top N highest probabilities in the probability set as target sub-agents, thus obtaining at least one target sub-agent. Alternatively, the main agent identifies the sub-agents corresponding to probabilities greater than or equal to a probability threshold as target sub-agents, also obtaining at least one target sub-agent.

[0172] N and the probability threshold are set by technicians according to the actual situation, and this application embodiment does not limit them.

[0173] The following explains how the main intelligent agent stores information in the long-term memory and fault memory.

[0174] In one possible implementation, the intelligent system includes a long-term memory and a fault memory. The main agent retrieves multiple task execution information from the agent execution log. The main agent encodes the multiple task execution information to obtain the task execution characteristics of each task. Based on the task execution characteristics, the main agent classifies the multiple task execution information to obtain the log type of each task execution information, which is either long-term memory or fault memory. Based on the log type of each task execution information, the main agent stores the multiple task execution information in the long-term memory or the fault memory respectively.

[0175] The task execution information consists of log data generated by the intelligent system during task execution. This log data can be stored as long-term memory or short-term memory in a long-term memory bank or a fault memory bank, respectively, for reference during subsequent task execution. Additionally, if no relevant information is retrieved from the long-term memory bank or the fault memory bank, a null value can be returned to avoid the impact of error messages on task planning.

[0176] For example, the main agent retrieves multiple task execution information from the agent execution log. The main agent encodes this information using an attention mechanism, obtaining task execution features for each task. The main agent performs a fully connected and normalized process on these task execution features to obtain classification values ​​for each task. The main agent identifies task execution information with classification values ​​greater than or equal to a classification threshold as long-term memory, and identifies task execution information with classification values ​​less than the threshold as fault memory. The main agent stores task execution information with long-term memory logs in the long-term memory repository, and stores task execution information with fault memory logs in the fault memory repository.

[0177] The classification threshold is set by technicians based on the actual situation, and the typical range is (0.3~1), with 0.9 being a suitable value. The above classification process can be implemented using a memory classifier, which can be a Bert-Based classifier, that is, a classifier composed of a Bert encoder and classification units.

[0178] In some embodiments, before classifying multiple task execution information, the main agent can preprocess the information. Preprocessing includes automated cleaning based on a rule engine, log deduplication, timestamp correction, and data structuring (e.g., converting to JSON format). The rule engine is configured by technical personnel according to requirements, and log deduplication can be implemented using the SimHash algorithm (threshold range 5-9). This preprocessing process can be implemented through an intelligent preprocessing layer. Furthermore, the state information of the sub-agents, generated during task execution, can also be preprocessed using the above methods and then classified using a memory classifier before being stored in short-term memory. When classifying state information, the memory classifier can either directly determine the log type as short-term memory based on its source, or it can use the same method as for long-term and short-term memory, determining the type of state information using the classification value. This application does not limit this approach. In other words, the three-level memory system supports both offline data (task execution information) and online data (state information) storage, meeting the diverse needs of intelligent systems when executing target tasks. A three-level memory system can be a relational database, a graph database, a vector database, etc. The log type can also be determined based on task identifier, device type, or geographic type; in this case, the data type stored in the memory system of the three-level database can be adjusted accordingly.

[0179] Through the aforementioned three-level memory system, it is possible to maintain a dynamic understanding and summary of historical information across time periods, forming a complete memory of knowledge, which is more helpful in understanding complex tasks.

[0180] In some embodiments, for the three-level memory system of an intelligent system—long-term memory (LM), fault memory (FM), and short-term memory (SM)—the main intelligent agent can dynamically clean up the three-level memory system to remove redundant data. Dynamic cleaning is typically based on time as the criterion; that is, it automatically cleans up information stored over a period of time. Since the information stored in different memory banks has different characteristics, the corresponding time periods for different memory banks are different. Generally speaking, the time T1 corresponding to the long-term memory is greater than or equal to the time T2 corresponding to the fault memory, and the time T2 corresponding to the fault memory is greater than the time T3 corresponding to the short-term memory. Taking T1 as 30 days, T2 as 30 days, and T3 as 1 hour as an example, information stored 30 days ago in the long-term and fault memory banks will be automatically cleaned up, and information stored 1 hour ago in the short-term memory will be automatically cleaned up. Of course, the values ​​of T1, T2, and T3 can be set by those skilled in the art according to actual conditions, and this application embodiment does not limit this. In addition to dynamic cleanup, a weighted approach can be used to process data in a three-level memory system. For example, information closer to the current time can be given a higher weight, while information farther away from the current time can be given a lower weight. Information with a higher weight is easier to retrieve. In this way, the timeliness of information can be maintained while preserving its integrity.

[0181] By dynamically cleaning the three-level memory system as described above, the information stored in the system can be kept up-to-date, which helps in executing the target task and avoids the adverse effects of outdated data on task planning. The process of memory classification and dynamic cleaning described above can be found in [reference needed]. Figure 5 The process is shown below.

[0182] 307. Based on the task description information of the target task, the second multimodal information collected by the multimodal sensors, and the fault reference information, the main intelligent agent re-plans the target task to obtain the second task planning information.

[0183] The replanning of the target task can be performed either directly after a fault occurs, or it can be performed after the fault severity is determined and the fault level is greater than or equal to a first fault level threshold. In other words, replanning of the target task is only performed when the fault severity is high. For example, selectable fault levels include 0 (extremely severe), 1 (severe), and 2 (warning). If the fault level is 0, replanning of the target task can be performed directly. If the fault level is 1 or 2, partial control of the sub-agents can be performed first, as described in step 306 above, followed by subsequent judgments. Alternatively, the sub-agents can combine fault information to replan the sub-tasks; this embodiment does not limit this approach.

[0184] In one possible implementation, the main agent inputs the second multimodal information and fault reference information into a large multimodal model. The large multimodal model encodes the second multimodal information and fault reference information to obtain the second multimodal features of the second multimodal information and the fault reference features of the fault reference information. The main agent then fuses the task description features, the second multimodal features, and the fault reference features of the task description information using the large multimodal model to obtain the second multimodal fused features. Finally, the main agent decodes the second multimodal fused features using the large multimodal model to obtain the second task planning information.

[0185] During the execution of the target task, the environmental conditions around the intelligent system may change. For example, in the case of a single robot, the robot's movement will alter the surrounding environment. Therefore, during the re-planning of the target task, the second multimodal information collected by multimodal sensors will be used to obtain the latest environmental conditions around the intelligent system. The task description features of the task description information have already been extracted in step 302, so there is no need to repeat the feature extraction using the multimodal large model; the task description features obtained in step 302 can be used directly. Referring to the relevant description in step 302, after a fault occurs, the second multimodal information and fault reference information are input into the multimodal large model through the emergency channel. The processing priority of information input through the emergency channel is higher than that of the regular input channel, thus allowing the fault to be handled preferentially.

[0186] In this implementation, the main intelligent agent uses the second multimodal information, fault reference information, and task description information to re-plan the target task and obtain the second task planning information. The second task planning information is the task planning information obtained after re-planning the target task. Since it incorporates fault reference information, it helps to continue executing the target task when the fault is eliminated, thereby better completing the target task.

[0187] To provide a clearer explanation of the above embodiments, the following description is divided into several parts.

[0188] The first part involves inputting the second multimodal information and fault reference information of the main intelligent agent into a large multimodal model. The large multimodal model encodes the second multimodal information and fault reference information to obtain the second multimodal features of the second multimodal information and the fault reference features of the fault reference information.

[0189] In one possible implementation, the main agent inputs the second multimodal information and fault reference information into a large multimodal model. The fault reference information is then encoded using the text encoder of the large multimodal model based on an attention mechanism to obtain fault reference features. The main agent then encodes the second multimodal information using the multimodal encoder of the large multimodal model based on an attention mechanism to obtain the second multimodal features of the second multimodal information.

[0190] It should be noted that the above encoding process can be executed either sequentially or synchronously, and this application embodiment does not limit this.

[0191] In the second part, the main intelligent agent fuses the task description features, second multimodal features, and fault reference features of the task description information through a multimodal large model to obtain the second multimodal fusion features.

[0192] In one possible implementation, the main agent uses a multimodal large model to fuse the task description features, second multimodal features, and fault reference features of the task description information based on a cross-attention mechanism to obtain the second multimodal fused features.

[0193] In this implementation, a cross-attention mechanism is used to fuse the task description features, second multimodal features, and fault reference features of the task description information, thereby fully modeling the task description features, second multimodal features, and fault reference features of the task description information and obtaining a second multimodal fusion feature with strong expressive power.

[0194] The above implementation method is illustrated below with two examples.

[0195] Example 1: The main agent uses a multimodal large model and a cross-attention mechanism to fuse task description features and second multimodal features to obtain a third fused feature. The main agent also uses a multimodal large model and a cross-attention mechanism to fuse the third fused feature and fault reference features to obtain a second multimodal fused feature.

[0196] In the first fusion process described above, the task description feature serves as the query, while the second multimodal feature provides the key and value. In the second fusion process, the third fusion feature serves as the query, while the fault reference feature provides the key and value. Providing the key and value means multiplying the feature by the key transformation matrix and the value transformation matrix, respectively. Both the key transformation matrix and the value transformation matrix are linear transformation matrices obtained through training.

[0197] Additionally, it should be noted that the above description is based on the example of fusing the task description feature and the second multimodal feature during the first fusion. In other possible implementations, any two other features can be fused during the first fusion, and the remaining feature can be fused during the second fusion. This application does not limit this.

[0198] Example 2: The main agent uses a multimodal large model and a cross-attention mechanism to fuse task description features and second multimodal features to obtain a third fused feature. The main agent uses the multimodal large model and a cross-attention mechanism to fuse task description features and fault reference features to obtain a fourth fused feature. The main agent uses the multimodal large model and a cross-attention mechanism to fuse second multimodal features and fault reference features to obtain a fifth fused feature. The main agent concatenates the third, fourth, and fifth fused features to obtain the second multimodal fused feature.

[0199] The third part involves the main intelligent agent decoding the second multimodal fusion features through the multimodal large model to obtain the second task planning information.

[0200] In one possible implementation, the main agent uses a multimodal large model and an attention mechanism to perform multiple rounds of iterative decoding on the second multimodal fusion features to obtain the second task planning information.

[0201] The second task planning information includes action execution information of multiple sub-agents. The action execution information is used to indicate the actions that the sub-agents need to perform and the timing of the action execution. For example, the multiple action execution information in the second task planning information are stored in a certain order, and the order of the action execution information in the second task planning information can be used to indicate the timing of the action execution indicated by the action execution information.

[0202] 308. The main agent sends the second task planning information to multiple sub-agents.

[0203] In this process, after the second task planning information is sent to multiple sub-agents, the multiple sub-agents can perform subsequent control based on the second task planning information.

[0204] The following is combined with Figure 6The technical solutions provided in the embodiments of this application will be described.

[0205] See Figure 6 Multimodal sensors, including microphones, cameras, and radar, collect multimodal information, including audio, images, and point clouds. The multimodal sensors send this information to a preprocessing module for preprocessing. The preprocessing module then sends the preprocessed information to the main agent. During normal task planning, the main agent retrieves task reference information from long-term memory. Using the acquired multimodal information, task description information (obtained through speech recognition of the audio), and the task reference information, the main agent performs task planning, obtaining the first task planning information—a process known as passive retrieval. The main agent sends this first task planning information to multiple sub-agents. These sub-agents, based on the first task planning information and state information retrieved from short-term memory, execute corresponding actions to collaboratively complete the designated task. During the execution of actions, the sub-agents store their state information in real-time in short-term memory, enabling information sharing with other sub-agents. In the event of a malfunction in the intelligent system, the primary agent retrieves fault reference information from the fault memory. Using the acquired multimodal information, task description information (obtained through speech recognition of the audio), and the fault reference information, the primary agent re-plans the task, obtaining second task planning information. This process is also known as the active retrieval mode. The primary agent sends the second task planning information to multiple sub-agents. These sub-agents, based on the second task planning information and state information retrieved from the short-term memory, execute corresponding actions to collaboratively complete the designated task while eliminating the fault.

[0206] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0207] The technical solution provided in this application embodiment allows the main intelligent agent to perform task planning based on the task description information of the target task, the first multimodal information from the multimodal sensor, and the task reference information obtained from the long-term memory, thus obtaining first task planning information. This process incorporates the task reference information, resulting in higher accuracy of the obtained first task planning information. The main intelligent agent sends the first task planning information to multiple sub-intelligent agents. Each sub-intelligent agent executes corresponding actions based on the state information obtained from the short-term memory and the first task planning information. Because the sub-intelligent agents incorporate state information when executing actions, the actions executed by multiple sub-intelligent agents are less likely to conflict, leading to better completion of the target task.

[0208] This application also provides an intelligent system, including a main intelligent agent and multiple sub-intelligent agents.

[0209] The main intelligent agent is used to perform task planning for the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory bank, and obtain the first task planning information.

[0210] The master agent is also used to send the first task planning information to multiple sub-agents.

[0211] Each sub-agent is used to obtain state information from the short-term memory and execute corresponding actions based on the state information and the first task planning information to cooperate in completing the target task. The state information is used to represent the agent state of multiple sub-agents.

[0212] In one possible implementation, the main intelligent agent is used to encode the task description information, the first multimodal information, and the task reference information through a multimodal large model to obtain the task description features of the task description information, the first multimodal features of the first multimodal information, and the task reference features of the task reference information.

[0213] The main intelligent agent is used to fuse task description features, first multimodal features, and task reference features through a multimodal large model to obtain the first multimodal fused features.

[0214] The main intelligent agent is used to decode the first multimodal fusion features through the multimodal large model to obtain the first task planning information.

[0215] In one possible implementation, the main agent is used to encode task description information and task reference information based on an attention mechanism using a text encoder of a multimodal large model to obtain task description features of the task description information and task reference features of the task reference information, and to encode first multimodal information based on an attention mechanism using a multimodal encoder of a multimodal large model to obtain first multimodal features of the first multimodal information.

[0216] The main intelligent agent is used to perform multiple rounds of iterative decoding of multimodal fusion features based on an attention mechanism through a large multimodal model to obtain the first task planning information.

[0217] In one possible implementation, the main agent is used to query associated task description information in a long-term memory based on task description information.

[0218] The main intelligent agent is used to populate the task description information and related task description information into the prompt text template to obtain the search-enhanced prompt text.

[0219] The main agent is used to input the search enhancement suggestion text into the search enhancement model, and the search enhancement model processes the search enhancement suggestion text to obtain task reference information.

[0220] In one possible implementation, for any one of the multiple sub-agents, state information is obtained from the short-term memory and sub-planning information corresponding to the sub-agent is obtained from the first task planning information.

[0221] Sub-agents are used to generate action execution information based on state information and sub-planning information.

[0222] A sub-agent is used to perform actions indicated by action execution information.

[0223] In one possible implementation, a sub-agent is used to input state information and sub-planning information into an action execution information generation model, and to encode the state information and sub-planning information through the action execution information generation model to obtain action execution features.

[0224] The sub-agent is used to generate a model from action execution information, decode the action execution features, and obtain the action execution information.

[0225] In one possible implementation, the sub-agent is also used to synchronize the agent's state to a short-term memory bank in real time.

[0226] In one possible implementation, the main intelligent agent is used to retrieve fault reference information from a fault memory based on the fault information when a fault occurs during the execution of the target task by the intelligent system.

[0227] The main intelligent agent is used to re-plan the target task based on the task description information of the target task, the second multimodal information collected by the multimodal sensors, and the fault reference information, so as to obtain the second task planning information.

[0228] The master agent is used to send the second task planning information to multiple sub-agents.

[0229] In one possible implementation, the main intelligent agent is used to determine the fault level in response to a fault occurring when the intelligent system is performing a target task. The fault level is used to indicate the severity of the fault.

[0230] The master agent is used to send control commands to at least one of the multiple sub-agents based on the fault level. The control commands are used to control the sub-agent to stop executing the corresponding actions.

[0231] In one possible implementation, the master agent is configured to send control commands to multiple sub-agents when the fault level is greater than or equal to a first fault level threshold. When the fault level is less than the first fault level threshold, the master agent is configured to determine at least one target sub-agent from the multiple agents based on fault information, task reference information, and task description information of the target task. The master agent then sends control commands to the at least one target sub-agent.

[0232] In one possible implementation, the main agent is used to input the second multimodal information and fault reference information into a large multimodal model, and encode the second multimodal information and fault reference information through the large multimodal model to obtain the second multimodal features of the second multimodal information and the fault reference features of the fault reference information. The main agent is also used to fuse the task description features, the second multimodal features, and the fault reference features of the task description information through the large multimodal model to obtain the second multimodal fused features. Finally, the main agent decodes the second multimodal fused features through the large multimodal model to obtain the second task planning information.

[0233] In one possible implementation, the intelligent system includes a long-term memory and a fault memory. A master agent retrieves multiple task execution information from the agent's execution log. The master agent encodes the multiple task execution information to obtain the task execution characteristics of each task. The master agent classifies the multiple task execution information based on its task execution characteristics to obtain the log type for each task, which is either long-term memory or fault memory. The master agent stores the multiple task execution information in either the long-term memory or the fault memory, based on their respective log types.

[0234] It should be noted that the agent-based task planning device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the intelligent system can be divided into different functional modules to complete all or part of the functions described above. In addition, the agent-based task planning device and the agent-based task planning method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0235] The technical solution provided in this application embodiment allows the main intelligent agent to perform task planning based on the task description information of the target task, the first multimodal information from the multimodal sensor, and the task reference information obtained from the long-term memory, thus obtaining first task planning information. This process incorporates the task reference information, resulting in higher accuracy of the obtained first task planning information. The main intelligent agent sends the first task planning information to multiple sub-intelligent agents. Each sub-intelligent agent executes corresponding actions based on the state information obtained from the short-term memory and the first task planning information. Because the sub-intelligent agents incorporate state information when executing actions, the actions executed by multiple sub-intelligent agents are less likely to conflict, leading to better completion of the target task.

[0236] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the agent-based task planning method in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0237] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. The processor of the intelligent system reads the program code from the computer-readable storage medium and executes the program code, causing the intelligent system to perform the above-described agent-based task planning method.

[0238] In some embodiments, the computer program involved in the present application embodiments may be deployed on a smart system for execution, or executed on multiple smart systems located in one location, or executed on multiple smart systems distributed in multiple locations and interconnected through a communication network. Multiple smart systems distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0239] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0240] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A task planning method based on intelligent agents, characterized in that, Applied to an intelligent system, the intelligent system comprising a main intelligent agent and multiple sub-intelligent agents, the method includes: The main intelligent agent performs task planning on the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory bank, and obtains the first task planning information. The main intelligent agent sends the first task planning information to multiple sub-intelligent agents; Each of the sub-agents obtains state information from the short-term memory and performs corresponding actions based on the state information and the first task planning information to cooperate in completing the target task. The state information is used to represent the agent state of the multiple sub-agents. The main intelligent agent performs task planning on the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained from the long-term memory, to obtain first task planning information, including: The main intelligent agent encodes the task description information, the first multimodal information, and the task reference information using a multimodal large model to obtain the task description features of the task description information, the first multimodal features of the first multimodal information, and the task reference features of the task reference information. The main intelligent agent fuses the task description features, the first multimodal features, and the task reference features through the multimodal large model to obtain the first multimodal fusion feature; The main agent decodes the first multimodal fusion feature through the multimodal large model to obtain the first task planning information. The first task planning information includes sub-planning information corresponding to multiple sub-agents, and the sub-planning information is used to indicate the sub-tasks assigned to the corresponding sub-agents.

2. The method according to claim 1, characterized in that, The main agent encodes the task description information, the first multimodal information, and the task reference information using the multimodal large model to obtain the task description features of the task description information, the first multimodal features of the first multimodal information, and the task reference features of the task reference information, including: The main agent encodes the task description information and the task reference information using the text encoder of the multimodal large model based on an attention mechanism to obtain the task description features of the task description information and the task reference features of the task reference information. It also encodes the first multimodal information using the multimodal encoder of the multimodal large model based on an attention mechanism to obtain the first multimodal features of the first multimodal information.

3. The method according to claim 1, characterized in that, The main intelligent agent performs task planning for the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory. Before obtaining the first task planning information, the method further includes: The main intelligent agent queries the long-term memory for related task description information based on the task description information. The main intelligent agent fills the task description information and the associated task description information into the prompt text template to obtain the search-enhanced prompt text; The main agent inputs the search enhancement prompt text into the search enhancement model, and processes the search enhancement prompt text through the search enhancement model to obtain the task reference information.

4. The method according to claim 1, characterized in that, Each of the sub-agents retrieves state information from a short-term memory and performs corresponding actions based on the state information and the first task planning information, including: For any one of the plurality of sub-agents, the sub-agent obtains state information from the short-term memory and obtains the sub-planning information corresponding to the sub-agent from the first task planning information; The sub-agent generates action execution information based on the state information and the sub-planning information; The sub-agent performs the action indicated by the action execution information.

5. The method according to claim 4, characterized in that, The sub-agent generates action execution information based on the state information and the sub-planning information, including: The sub-agent inputs the state information and the sub-planning information into the action execution information generation model, and encodes the state information and the sub-planning information through the action execution information generation model to obtain action execution features; The sub-agent decodes the action execution features through the action execution information generation model to obtain the action execution information.

6. The method according to claim 4 or 5, characterized in that, The method further includes: The sub-agent synchronizes the agent's state to the short-term memory in real time.

7. The method according to claim 1, characterized in that, The method further includes: In response to a fault occurring when the intelligent system is performing the target task, the main intelligent agent retrieves fault reference information from the fault memory based on the fault information of the fault. The main intelligent agent re-plans the target task based on the task description information of the target task, the second multimodal information collected by the multimodal sensor, and the fault reference information, to obtain the second task planning information. The master agent sends the second task planning information to the multiple sub-agents.

8. The method according to claim 7, characterized in that, The method further includes: In response to a failure occurring when the intelligent system is performing the target task, the main intelligent agent determines the failure level, which is used to indicate the severity of the failure; Based on the fault level, the master agent sends a control command to at least one of the plurality of sub-agents, the control command being used to control the sub-agent to stop executing the corresponding action.

9. An intelligent system, characterized in that, The intelligent system includes a main intelligent agent and multiple sub-intelligent agents; The main intelligent agent is used to perform task planning for the target task based on the task description information of the target task, the first multimodal information collected by the multimodal sensor, and the task reference information obtained through the long-term memory bank, to obtain the first task planning information. The main intelligent agent is also used to send the first task planning information to multiple sub-intelligent agents; Each of the sub-agents is used to obtain state information from the short-term memory bank and to perform corresponding actions based on the state information and the first task planning information in order to cooperate in completing the target task. The state information is used to represent the agent state of the multiple sub-agents. Specifically, the main agent is used to encode the task description information, the first multimodal information, and the task reference information using a multimodal large model to obtain task description features of the task description information, first multimodal features of the first multimodal information, and task reference features of the task reference information; to fuse the task description features, the first multimodal features, and the task reference features using the multimodal large model to obtain a first multimodal fusion feature; and to decode the first multimodal fusion feature using the multimodal large model to obtain the first task planning information, which includes sub-planning information corresponding to multiple sub-agents, and the sub-planning information is used to indicate the sub-tasks assigned to the corresponding sub-agents.

Images