Multi-modal task understanding and collaborative planning method and system oriented to agent with body

Abstract

The invention relates to the technical field of robot task execution, and discloses a multi-modal task understanding and collaborative planning method and system for an agent with a body, and the method comprises the steps: S1, obtaining the GPS spatial positioning data and visual sensor data of the agent with the body in real time, recognizing a semantic feature object in an environment through a target detection model, and carrying out the recognition of the semantic feature object in the environment; fusing the spatial positioning data and the visual semantic features and converting the fused spatial positioning data and visual semantic features into structured text description; according to the method, environmental physical information and visual semantics are fused into a structured text through multi-modal context sensing, and a user composite instruction is dynamically deconstructed into a standard function original intonation sequence by taking a function call mechanism of a large language model as a center; and self-adaptive execution of the task is realized through state feedback and real-time re-planning in the execution process, so that the defects of semantic understanding fault, execution logic stiffness, non-uniform interface calling and the like in the prior art are effectively overcome.

Description

Technical Field

[0001] This application relates to the technical field of robot task execution, and in particular to a method and system for multimodal task understanding and collaborative planning for embodied intelligent agents. Background Technology

[0002] Embodied intelligent agents are intelligent systems capable of perceiving their environment through sensors and acting upon it through actuators. Typical applications include home service robots, commercial delivery robots, autonomous vehicles, and intelligent warehousing equipment. With the rapid development of artificial intelligence, computer vision, and natural language processing technologies, multimodal perception and understanding capabilities are gradually becoming key supports for embodied intelligent agents to achieve autonomous decision-making and human-machine collaboration. Current research and practice generally focus on how to fuse multi-source sensor information such as vision, LiDAR, and GPS to construct a spatial semantic representation of the environment. At the same time, natural language human-computer interaction is evolving from simple keyword commands to complex intent understanding, requiring intelligent agents to complete continuous task sequences across scenarios and devices under unstructured commands. Against this background, task decomposition, resource scheduling, and dynamic execution mechanisms oriented towards high-level semantics have become core technical links for improving the autonomy and generalization capabilities of embodied intelligent agents.

[0003] However, existing embodied intelligent agent systems still have several limitations in task understanding and collaborative planning. First, most systems' ability to parse user instructions is limited to preset keywords or fixed sentence structures, making it difficult to effectively handle complex natural language instructions containing fuzzy spatial references, implicit intentions, or multiple dependencies, resulting in a significant gap between semantic understanding and physical execution. Second, task execution processes are usually pre-programmed by humans into fixed action sequences, lacking real-time response mechanisms to dynamic environmental changes, execution anomalies, or temporary changes in user instructions, resulting in insufficient system robustness and scenario adaptability. In addition, functional modules such as motion control, device control, and interactive feedback often belong to heterogeneous software stacks with inconsistent interface definitions and a lack of a unified scheduling framework that can map high-level semantic intentions to low-level atomic operations, making the development and expansion of cross-modal and cross-device collaborative tasks highly complex. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, such as semantic understanding gaps, rigid execution logic, and inconsistent interface calls, this application provides a method and system for multimodal task understanding and collaborative planning for embodied intelligent agents.

[0005] Firstly, this application provides a method for multimodal task understanding and cooperative planning for embodied intelligent agents, employing the following technical solution: A method for multimodal task understanding and cooperative planning for embodied intelligent agents, the method comprising: S1. Real-time acquisition of GPS spatial positioning data and visual sensor data of the embodied intelligent agent; identification of semantic features in the environment through a target detection model; mapping of the GPS spatial positioning data to preset spatial hierarchical semantic labels; mapping of the two-dimensional bounding box obtained by target detection of the image collected by the visual sensor to the three-dimensional spatial position in the robot coordinate system through a camera imaging geometric model; construction of a spatial semantic dynamic mapping table with spatial hierarchical semantic labels and visual object categories as joint indexes; dynamic association of physical coordinates; calculation of the current optimal mapping value based on visual detection confidence and historical access frequency; and conversion of the fusion result into a structured text description. S2. The motion control capability, IoT device control capability, and human-computer interaction feedback capability of the embodied intelligent agent are encapsulated into function primitives with standardized input and output interfaces, and a function toolbox that can be directly called by large language models is constructed. S3. Pre-construct an environmental scene knowledge graph. The nodes of the knowledge graph represent entity objects or regions in the environment, and the edges represent spatial inclusion relationships, functional association relationships, or operational dependency relationships between entities. The structured text description and the user's natural language instructions are concatenated to form a prompt input to the large language model. The large language model selects at least one function primitive from the function toolbox through its function call mechanism, and uses the subgraph structure of the knowledge graph as reasoning constraints. Based on the current environmental context and the instruction intent, it autonomously reasones the task dependency relationship and outputs a function call sequence composed of the selected function primitives ordered according to the execution logic. S4. Parse the function call sequence, and the scheduling execution engine calls the corresponding function primitives in sequence. During the function execution, continuously monitor the changes in the environment state and the feedback of the execution results. When an execution failure is detected or the environment state does not match the preset preconditions, real-time replanning is triggered, and the large language model regenerates the function call sequence adapted to the current state.

[0006] By adopting the above technical solutions, a four-step architecture of multimodal perception, function abstraction, large language model decomposition, and dynamic execution was constructed. The raw sensor data was transformed into structured text, and the function call mechanism of the large language model was used to realize the end-to-end mapping from compound natural language instructions to structured function call sequences. An execution feedback and replanning loop was established, which solved the technical problems of semantic understanding gaps and rigid execution logic in traditional systems.

[0007] Optionally, in step S1, the process of fusing spatial positioning data with visual semantic features and transforming it into a structured text description includes: Map GPS spatial positioning data to preset spatial hierarchical semantic labels; The image acquired by the vision sensor is input into a lightweight object detection network, which outputs the object categories in the environment and their two-dimensional bounding boxes. Based on the camera imaging geometric model, the two-dimensional bounding box is mapped to its three-dimensional spatial position in the robot coordinate system; Construct a spatial semantic dynamic mapping table, which uses spatial hierarchical semantic labels and visual object categories as a joint index to dynamically associate the corresponding physical coordinates or relative positions; When the same semantic label corresponds to multiple physical coordinates, the current optimal mapping value is calculated based on the visual detection confidence and historical access frequency.

[0008] By adopting the above technical solution, the deep fusion of GPS spatial hierarchical labels and visual semantic features is realized in step S1. Through the geometric mapping from two-dimensional bounding boxes to three-dimensional coordinates and the construction of a spatial semantic dynamic mapping table, the embodied intelligent agent can dynamically associate fuzzy semantics with physical coordinates without the need for manual preset of all position mapping relationships.

[0009] Optionally, the process of calculating the current optimal mapping value includes: Define semantic tags The corresponding set of candidate physical coordinates is ,in, , For physical coordinate vectors, For visual detection confidence, For timestamps; Then the current optimal mapping coordinates Determined by the following formula: in, For the current time, The time decay constant, This is a normalization function for the frequency of historical visits. , , Preset weighting coefficients; When set When the coordinates are empty or the confidence of all candidate coordinates is below a preset threshold, an active visual search primitive is triggered to supplement the mapping data.

[0010] By adopting the above technical solution, a weighted decision mechanism that integrates visual detection confidence, time decay factor and historical access frequency is introduced into the spatial semantic dynamic mapping. When the same semantic label corresponds to multiple candidate coordinates, the optimal mapping value can be selected adaptively, and visual search can be actively triggered when mapping data is missing, which significantly improves the accuracy and robustness of semantic localization.

[0011] Optionally, in step S2, the function primitives include at least the following categories: The location movement primitive takes target semantic points, navigation strategy patterns or path planning constraints as input parameters, calls the underlying navigation stack and returns the execution status and current position feedback. Device control primitives, whose input parameters include device type, device identifier, control action and action parameters, send instructions directly to smart home gateways or devices via IoT protocols; Voice interaction primitives, whose input parameters include the text to be read, speech rate or timbre, call the speech synthesis engine to convert the text into an audio stream and play it through the speaker; Environmental perception primitives take the region of interest or target object category as input parameters and output a Boolean value or metric value of environmental state as output. The standardized interfaces of the function primitives are defined using a unified description language and registered to the tool call interface layer of the large language model.

[0012] By adopting the above technical solution, four types of capabilities—motion control, device control, voice interaction, and environmental perception—are encapsulated into function primitives with standardized input and output interfaces. These primitives are then registered to the large language model tool call layer using a unified description language. This solves the technical problems of inconsistent interfaces in heterogeneous software stacks and the inability of high-level semantics to directly map low-level instructions.

[0013] Optionally, the process of generating the function call sequence is constrained and guided based on the scene graph; An environmental scene knowledge graph is pre-constructed, where nodes represent entity objects or regions in the environment, and edges represent spatial inclusion relationships, functional association relationships, or operational dependencies between entities. The knowledge graph is generated through manual annotation, import of building information models, or machine autonomous exploration and mapping, and supports dynamic updates; The structured text description and user instructions are mapped to a subset of nodes in the knowledge graph, thus decomposing the task into a path search problem on the knowledge graph. The large language model uses the subgraph structure of the knowledge graph as reasoning constraints to generate a function call sequence that conforms to spatial logic and functional logic.

[0014] By adopting the above technical solution, a scene knowledge graph is introduced as a reasoning constraint in the function call sequence generation process. The task is decomposed into a path search problem on the graph, so that the call sequence generated by the large language model conforms to the spatial inclusion relationship and functional dependency relationship between entities, avoiding logical conflicts or physically unexecutable action sequences.

[0015] Optionally, in step S4, the process of triggering real-time replanning includes: Define an execution state machine, including four states: waiting to execute, executing, executing successfully, and executing unsuccessfully. Establish a set of preconditions and a set of postconditions for function primitives. The preconditions and postconditions are described in predicate logic or key-value pair form and are consistent with the environment state representation model. When a function primitive is executed, if the precondition does not match the current environment state, or the execution result is inconsistent with the postcondition, the execution will be interrupted. The sequence of function primitives that have not been executed after the breakpoint, the cumulative effect of the currently executed primitives, and the latest environmental awareness information are concatenated into a replanning prompt, which is then input into the large language model. The large language model generates a sequence of supplementary or corrective function calls that converges from the current interrupted state to the target task.

[0016] By adopting the above technical solution, a set of preconditions and postconditions for the execution state machine and function primitives is established. When the execution conditions do not match or the result is abnormal, the execution is automatically interrupted, and the interruption context is concatenated into a large language model for replanning prompts. This realizes the dynamic task reconstruction capability that converges from the current state to the target task.

[0017] Optionally, in step S4, the process of the large language model regenerating a function call sequence adapted to the current state includes: Candidate function primitives are sorted according to task refactoring priority; The task reconstruction priority is obtained by weighted summation of multiple evaluation indicators; The evaluation metrics include: the progress gain of the environment state relative to the target state after executing the candidate function primitive, the semantic dependency strength between the candidate function primitive and the target task, and the reciprocal of the estimated time or energy cost of executing the candidate function primitive. The weighting coefficients used in the weighted summation are adaptively adjusted according to the current scenario, including increasing the weight of the reciprocal cost index when the task is urgent, and increasing the weight of the reciprocal cost index to prioritize energy saving when the robot's battery is low. Candidate function primitives are selected in descending order of task reconstruction priority until all preconditions and post-effect constraints of the target task are met.

[0018] By adopting the above technical solution, a task reconstruction priority evaluation mechanism is introduced in the replanning process. The candidate function primitives are ranked by a weighted index of three dimensions: progress gain, semantic dependency strength, and execution cost. The weight coefficients can be adaptively adjusted according to the scenario, enabling the large language model to select the best option from multiple feasible solutions and optimize task recovery efficiency and resource consumption.

[0019] Optionally, the method further includes: When there are multiple compound instructions to be executed within the same scheduling period, the function call sequences corresponding to each compound instruction are merged into a global task queue. Based on the usage requirements of each function primitive for mobile channels, device resources, and computing resources, a resource conflict detection matrix is ​​established. The rows and columns of the matrix are each function primitive and each resource item, respectively, and the cells mark the usage requirements of the primitive for specific resources. For function primitives with resource conflicts, conflict resolution is performed based on instruction priority, user weight, or energy consumption constraints. The instruction priority ranges from 1 to 10, the user weight ranges from 0 to 1, and the energy consumption constraint is quantified based on the ratio of the task's estimated energy consumption to the robot's current remaining power, generating conflict-free parallel or serial execution plans.

[0020] By adopting the above technical solution, a global task queue and resource conflict detection matrix are established in multi-instruction concurrent scenarios. Conflicts are resolved based on priority, user weight or energy consumption constraints, and parallel or serial execution plans without resource contention are generated, thus solving the problems of resource contention and execution deadlock in multi-task scheduling.

[0021] Optionally, the method further includes: The function call sequence generated by the large language model is decomposed into multiple subtasks that can be executed in parallel or collaboratively based on spatial decoupling, device independence and functional dependencies. Each subtask is assigned to a different embodied intelligent agent execution terminal, and the allocation strategy takes into account the terminal hardware capabilities, current location and current load. Establish a lightweight communication protocol between the central coordinator and each execution terminal. The central coordinator maintains a global task schedule and a shared environment semantic map. When any execution terminal senses an environmental change or execution anomaly, it uploads the updated local state to the central coordinator. The central coordinator then triggers a global replanning or a local task migration. If it only involves a single subtask, a local replanning is triggered. If it affects the overall progress, a global replanning is triggered. If a terminal fails, the terminal's task is migrated to another available terminal.

[0022] By adopting the above technical solution, the function call sequence generated by the intelligent agent is decomposed into sub-tasks that can be executed in parallel and distributed to multiple execution terminals. A lightweight communication protocol and state synchronization mechanism between the central coordinator and the terminals are established, realizing distributed collaborative execution and dynamic task migration across robots, and expanding the system's service capabilities in large-scale and complex scenarios.

[0023] Secondly, this application provides a multimodal task understanding and cooperative planning system for embodied intelligent agents, employing the following technical solution: A system for multimodal task understanding and collaborative planning for embodied intelligent agents, characterized in that it is used to implement the method for multimodal task understanding and collaborative planning for embodied intelligent agents as described in any one of the above claims, the system comprising: The multimodal perception module, deployed on the embodied intelligent agent platform, includes a GPS receiving unit, a visual acquisition unit, and a multimodal information fusion unit. It is used to acquire spatial positioning data and visual sensor data in real time, identify semantic features in the environment through a target detection model, and fuse the spatial positioning data and visual semantic features into a structured text description. The function primitive library module is used to store predefined standardized interface function primitives, which cover four types of capabilities: motion control, device control, voice interaction and environmental awareness. The function primitives are registered to the function toolbox for use. The large language model decision module is deployed on a cloud server or a local edge computing node. It has a built-in function call interface adaptation layer, which is used to receive the structured text description and user natural language instructions. It selects function primitives from the function toolbox through the reasoning mechanism constrained by the scene knowledge graph, autonomously reasones the task dependencies, and outputs the function call sequence composed of the selected function primitives ordered by the execution logic. The task scheduling and execution module includes a function call parser, an execution state machine, and a replanning trigger. It is used to parse the function call sequence, call the corresponding function primitives in sequence, monitor environmental state changes and execution result feedback in real time, and trigger a replanning request when execution failure is detected or the environmental state does not meet the preset preconditions. The collaborative communication module is used to establish a lightweight communication protocol between the central coordinator and each execution terminal in a multi-agent deployment scenario, so as to realize global task progress synchronization, maintenance of shared environmental semantic map, and status feedback and task scheduling of execution terminals.

[0024] By adopting the above technical solutions, a complete link is formed from environmental perception, semantic understanding, task decision-making to execution feedback, with five core modules: multimodal perception, function primitive library, large language model decision-making, task scheduling and execution, and collaborative communication. Among them, the large language model decision-making module and the task scheduling and execution module interact bidirectionally, providing real-time feedback on execution status and environmental changes, and supporting dynamic replanning and adaptive scheduling.

[0025] In summary, this application includes at least one of the following beneficial technical effects: This invention integrates environmental physical information and visual semantics into structured text through multimodal context awareness. Using a large language model's function call mechanism as the central element, it dynamically deconstructs user complex instructions into standardized function primitive call sequences. During execution, adaptive task execution is achieved through state feedback and real-time replanning. This effectively overcomes the shortcomings of existing technologies, such as semantic understanding gaps, rigid execution logic, and inconsistent interface calls. On one hand, by combining a function toolbox with large language model function calls, it achieves seamless conversion from fuzzy natural language to precise machine instructions, solving the problem of a single dimension of instruction understanding. On the other hand, through a dynamic replanning mechanism driven by execution feedback, the agent is freed from the constraints of fixed scripts and can respond to environmental changes and execution anomalies in real time. Simultaneously, standardized function primitives encapsulate heterogeneous execution capabilities, providing a unified scheduling framework for movement, operation, device linkage, and interactive feedback, significantly improving the task execution success rate and system scalability of the embodied agent in complex scenarios. Attached Figure Description

[0026] Figure 1 This is a flowchart of the steps of the multimodal task understanding and collaborative planning method for embodied intelligent agents disclosed in this invention.

[0027] Figure 2 This is a schematic diagram of the multimodal task understanding and collaborative planning system for embodied intelligent agents disclosed in this invention. Detailed Implementation

[0028] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0029] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0030] This application discloses a method for multimodal task understanding and collaborative planning for embodied intelligent agents, referring to... Figure 1 The method includes: S1. The embodied intelligent agent acquires large-scale spatial positioning data in real time through the vehicle-mounted GPS receiver unit to determine its current global location (such as building, floor, area). Simultaneously, it collects real-time images of the surrounding environment through visual sensors (such as monocular / binocular cameras, depth cameras), and inputs the collected images into a pre-trained lightweight target detection model to identify semantic features in the environment, including but not limited to furniture (beds, sofas, dining tables), appliances (air conditioners, televisions, lamps), passageways (doors, elevators, corridors), and other interactive objects. The system fuses GPS spatial positioning data with visual semantic features: on the one hand, it maps GPS coordinates to preset spatial hierarchical semantic labels (such as "first floor lobby" and "second floor bedroom"); on the other hand, it converts the two-dimensional bounding boxes output by target detection into three-dimensional spatial positions in the robot coordinate system through camera imaging geometry models. The fused result is organized into a structured text description, such as: "Current location: first floor living room; visual detection detected: sofa (coordinates x, y, z), air conditioner (coordinates x, y, z), door (coordinates x, y, z)". S2. The basic operational capabilities that the embodied intelligent agent can execute are atomized and abstracted to form function primitives with standardized input and output interfaces. The atomic capabilities cover three dimensions: motion control capabilities, including moving to a target semantic point, following a target, path planning and obstacle avoidance, etc.; IoT device control capabilities, including sending control commands to smart home devices through wireless communication protocols (such as Wi-Fi, Bluetooth, Zigbee), such as switching on / off, adjusting temperature, adjusting brightness, etc.; and human-computer interaction feedback capabilities, including voice synthesis broadcasting, facial expression or light status feedback, etc. Each function primitive is defined using a unified interface description language, clearly defining its name, parameter list, return value and calling constraints. All function primitives are registered in a centralized function toolbox, which provides a unified calling entry point and supports dynamically adding, updating or disabling function primitives. S3. The structured text description is concatenated with the user's input of the original natural language command to form a complete prompt text. This prompt text is input into the large language model (which can be deployed on a cloud server or a local edge computing node), and the model's function call function is enabled. The large language model first understands the explicit task goal (such as "deliver coffee to the bedroom") and implicit intent (such as "turn on the heater") contained in the user's command, and combines the current environmental context provided in the structured text description to autonomously infer the logical dependencies required to complete the task. On this basis, the model selects a set of relevant function primitives from the constructed function toolbox and organizes them into a structured function call sequence according to the execution order. S4. Parse the function call sequence. The scheduling execution engine calls the corresponding function primitives in sequence. Before each function primitive is executed, the system checks whether its preconditions (such as whether the movement path is unobstructed or whether the device is online) are met. During execution, the system continuously monitors changes in the environmental state and feedback on the execution results. After execution, the system verifies whether the post-effects are consistent with expectations. If the entire sequence is successfully executed, the task is completed. If it is detected that a function primitive fails to meet its preconditions or its execution result does not meet expectations due to environmental changes (such as path being blocked or device being offline), execution anomalies (such as capture failure), or user changes in instructions midway, the system immediately triggers a real-time replanning mechanism: the system packages the function primitive sequence that has not yet been executed after the current execution interruption point, the accumulated state generated by the executed primitives, and the latest collected environmental awareness information into a replanning prompt, and inputs it again into the large language model. The large language model regenerates a supplementary or corrected function call sequence adapted to the current scenario based on the current actual state and the original target task, and the system continues to execute the new sequence.

[0031] Through the above technical solution, this embodiment provides a multimodal task understanding and collaborative planning method for embodied intelligent agents. The method integrates environmental physical information and visual semantics into structured text through multimodal context awareness. Using the function call mechanism of a large language model as the central mechanism, it dynamically deconstructs user complex instructions into standardized function primitive call sequences. During execution, adaptive task execution is achieved through state feedback and real-time replanning. This effectively overcomes the shortcomings of existing technologies, such as semantic understanding gaps, rigid execution logic, and inconsistent interface calls. On the one hand, by combining the function toolbox with large language model function calls, a seamless conversion from fuzzy natural language to precise machine instructions is achieved, solving the problem of a single dimension of instruction understanding. On the other hand, through the dynamic replanning mechanism driven by execution feedback, the intelligent agent is freed from the constraints of fixed scripts and can respond to environmental changes and execution anomalies in real time. Simultaneously, standardized function primitives encapsulate heterogeneous execution capabilities, providing a unified scheduling framework for movement, operation, device linkage, and interactive feedback, significantly improving the task execution success rate and system scalability of embodied intelligent agents in complex scenarios.

[0032] In one embodiment, step S1, the process of fusing spatial positioning data with visual semantic features and converting it into a structured text description, includes: GPS spatial positioning data is mapped to preset spatial hierarchical semantic labels. Before running, the embodied intelligent agent will pre-establish a spatial hierarchical topology structure, such as associating semantic labels such as "first floor", "second floor", "living room", and "bedroom" with the corresponding GPS coordinate range or geofence. When the intelligent agent obtains real-time GPS coordinates, it can determine the current spatial semantic level by matching the coordinate range, such as "second floor corridor". This mapping relationship can be preset manually or automatically labeled through the SLAM mapping process. Images acquired by a visual sensor are input into a lightweight object detection network, which outputs the categories of objects in the environment and their two-dimensional bounding boxes. The lightweight object detection network can use models suitable for edge deployment, such as YOLO and MobileNet-SSD. After pre-training, it can recognize semantic features related to the task, such as beds, sofas, air conditioners, doors, and elevators. The network outputs the category label and bounding box coordinates in the image coordinate system for each detected object. Based on the camera imaging geometric model, the two-dimensional bounding box is mapped to the three-dimensional spatial position in the robot coordinate system. Specifically, using a pinhole camera model, combined with the camera's intrinsic parameters (focal length, principal point coordinates) and extrinsic parameters (camera's installation position and attitude relative to the robot base), the center point or bottom midpoint of the bounding box is back-projected into three-dimensional space. Combined with depth information (such as depth values ​​directly obtained by a depth camera, or distance estimated by monocular vision through ground assumptions), the three-dimensional coordinates of the target relative to the robot are obtained. Then, the robot odometry is used to convert them into physical coordinates in the global or local coordinate system. A dynamic spatial semantic mapping table is constructed. This mapping table uses spatial hierarchical semantic labels and visual object categories as a joint index to dynamically associate the corresponding physical coordinates or relative positions. The mapping table can be stored in an in-memory database. Each record contains: spatial label (e.g., "second-floor bedroom"), object category (e.g., "bed"), current optimal physical coordinates (x, y, z), historical candidate coordinate list, update timestamp, confidence score, and other information. When the agent detects the same category of object multiple times at the same spatial level, the mapping table will accumulate multiple candidate coordinates. When the same semantic label corresponds to multiple physical coordinates, the current optimal mapping value is calculated based on the visual detection confidence and historical access frequency.

[0033] The process of calculating the current optimal mapping value includes: Define semantic tags The corresponding set of candidate physical coordinates is ,in, , It is a three-dimensional physical coordinate vector. The confidence score for visual detection (ranging from 0 to 1, obtained by normalizing the classification probability of the object detection network; a higher value indicates a more reliable detection). The timestamp for the detection time; Then the current optimal mapping coordinates Determined by the following formula: in, The timestamp of the current moment is used to calculate the time decay factor. This is a time decay constant, typically set to 600 seconds (10 minutes) or 3600 seconds (1 hour), representing the effective duration of historical observations. Candidate coordinates exceeding this time threshold will have their weights decayed to near zero. This is a normalization function for historical access frequency, defined as the proportion of the cumulative number of times a candidate coordinate is selected as the optimal mapping value out of the total number of accesses. An exponential moving average can also be used for smoothing. , , The preset weighting coefficients can typically be set to [value]. =0.6 (focusing on the current detection confidence level) =0.3 (taking timeliness into account) =0.1 (incorporating historical preferences), the specific value can be determined through experimental calibration or adaptive learning; This fusion decision function integrates detection confidence, observation timeliness, and historical usage habits, enabling the agent to dynamically weigh evidence from different sources during semantic localization. For example, when the robot briefly glimpses a corner of the bed at the bedroom door, the confidence level is low, and the system tends to trust historical coordinates that have been successfully located multiple times before. However, when the bed's position is rearranged, the newly detected coordinates, due to their higher timeliness and sufficient confidence, will gradually replace the old coordinates. When set If the coordinates are empty (i.e., no object of this category has ever been detected in the area) or the confidence of all candidate coordinates is lower than a preset threshold (e.g., 0.3), the system determines that there is a lack of reliable semantic mapping data. At this time, an active visual search primitive is triggered, which drives the agent to perform a traversal scan in the current spatial hierarchy until the target object is detected and valid coordinates are obtained, thereby supplementing the mapping data.

[0034] In step S2, the function primitives include at least the following categories: Position movement primitives, such as `move_to(target, strategy, constraints)`, are defined as follows: `target` is the target semantic point, which can be a spatial semantic label (e.g., "bedroom"), physical coordinates, or a dynamically tracked target; `strategy` is the navigation strategy pattern, such as "shortest path," "energy-saving mode," or "obstacle avoidance priority"; and `constraints` are path planning constraints, such as "no carpeting areas" or "priority elevator use." This primitive calls the underlying navigation stack and returns the execution status (success / failure / in progress) and current position feedback. Device control primitives, such as `control_device(device_type, device_id, action, params)`, where `device_type` is the device type (air conditioner, light, curtains, etc.); `device_id` is the unique identifier of the device (such as MAC address, MQTT topic); `action` is the control action (set_power, set_temperature, set_brightness, etc.); and `params` is a dictionary of action parameters. This primitive sends commands directly to the smart home gateway or device via IoT protocols (such as MQTT, CoAP, HTTP) and returns operation confirmation results. Voice interaction primitives, such as speak(text, rate=1.0, voice='female'), where text is the text content to be read; rate is the speech rate multiplier (0.5~2.0); and voice is the voice selection. This primitive calls the speech synthesis engine to convert the text into an audio stream and play it through the speaker, and can be accompanied by emotional expressions or light feedback. Environmental awareness primitives, such as check_condition(region, object_category, attribute), where region is the region of interest (e.g., "near the dining table"); object_category is the category of the target object; and attribute is the status attribute to be queried (e.g., "temperature", "humidity", "whether there are people"). This primitive obtains the environmental status through vision, temperature and humidity sensors, or infrared detection and returns a Boolean value or a metric (e.g., 26.5℃). The standardized interfaces of all function primitives are defined using a unified description language (such as OpenAPI Schema or JSON Schema), which clearly defines the function name, parameter types, mandatory parameters, default values, return value format, and exception codes. These interface definitions are registered to the tool call interface layer of the large language model (such as OpenAI's functions parameters and LangChain's Tool abstraction), enabling the model to automatically select appropriate functions and fill in the parameters through natural language inference.

[0035] The process of generating the function call sequence is constrained and guided by the scene graph. An environmental scene knowledge graph is pre-constructed. This knowledge graph stores entities and relationships in the environment in a graph structure. Nodes represent physical entities or abstract regions, such as "first floor," "elevator," "bedroom A," "air conditioner A1," and "bed B1." Edges represent various relationships between entities, including: spatial inclusion relationships (such as "bedroom A ⊂ second floor" and "air conditioner A1 ⊂ bedroom A"); functional association relationships (such as "air conditioner A1 can adjust the temperature" and "bed B1 is used for resting"); and operational dependency relationships (such as "moving to bedroom A requires first entering the second floor" and "using the air conditioner requires first moving to bedroom A"). Knowledge graphs can be generated through manual annotation, import from Building Information Modeling (BIM), or autonomous machine exploration, and support dynamic updates; When the system retrieves a structured text description (such as "Currently in the second-floor corridor, the door of bedroom A has been detected") and combines it with a user instruction (such as "Go to bedroom A and turn on the air conditioner"), it maps the entities in the text description and the intent in the instruction to the corresponding subset of nodes in the knowledge graph. For example, "bedroom A" corresponds to the bedroom node in the graph, and "air conditioner" corresponds to the air conditioner device node associated with bedroom A. The task decomposition problem is transformed into a path search problem on a knowledge graph. The large language model no longer generates action sequences out of thin air. Instead, it starts from the current node (the graph node corresponding to the agent's current position) and ends at the target task node (such as "Air conditioner A1 - On"). It searches for an executable path on the graph. The edge sequence of this path corresponds to the function primitives to be executed and their dependency order. For example, the path is: "Currently in the second-floor corridor" → "Enter bedroom A" (requires move_to primitive) → "Turn on air conditioner A1" (requires control_device primitive). If there are branches or optional operations in the path, the model can select the optimal path according to the context. The large language model uses the subgraph structure of the knowledge graph as explicit reasoning constraints. The generated function call sequences naturally conform to spatial reachability, device dependency, and functional logic. For example, the model will not attempt to control the air conditioner directly without reaching the bedroom, nor will it incorrectly send TV control commands to the air conditioner. This method significantly reduces the probability of the model generating illegal or unexecutable sequences and improves the interpretability of task decomposition.

[0036] Through the above technical solution, this embodiment provides a task understanding and planning method for embodied intelligent agents. The method constructs a spatial semantic dynamic mapping table through multimodal fusion and introduces confidence-weighted decision-making to achieve adaptive semantic location. It encapsulates heterogeneous execution capabilities through standardized function primitives and registers them to the large language model tool layer. It guides the generation of function call sequences through scene knowledge graph constraints and forms a closed loop through execution feedback and dynamic replanning. This solves the problems of semantic understanding gaps, rigid execution logic, and inconsistent interface calls in the prior art. It achieves accurate mapping from fuzzy natural language to structured machine instructions, real-time adaptive adjustment of task sequences, and scalability for cross-scene and cross-device collaborative execution.

[0037] In one embodiment, step S4, triggering the real-time replanning process includes: The system defines an execution state machine. It establishes a state machine model for the execution process of each function primitive, which includes four basic states: waiting to execute (task sequence has been generated but not yet scheduled), executing (function primitive is calling underlying hardware or services), execution successful (primitive returns the expected result), and execution failed (primitive did not complete due to exception or returned an error code). The state machine continuously transitions during task execution, providing a basis for exception detection. Establish a set of preconditions and postconditions for function primitives. Preconditions refer to the environmental or system states that must be met before calling a function primitive. For example, the preconditions for the move primitive `move_to` may include "target point is reachable", "path is not blocked", and "robot is not currently charging". The preconditions for the device control primitive `control_device` may include "device is online", "device is in controllable mode", and "robot has arrived at the device's location". Postconditions refer to the expected changes to the environmental or system states after the function primitive is successfully executed. For example, the postcondition for the move primitive is "robot coordinates updated to the target point"; the postcondition for the device control primitive is "device state changed to the target state (e.g., air conditioner is turned on, temperature is set to 26℃)". Preconditions and postconditions can be described using predicate logic or key-value pairs and should be consistent with the environmental state representation model (e.g., semantic map, device state table). When the execution engine schedules a function primitive, it first checks whether its preconditions match the current environment state. For example, if `move_to` detects that the path is blocked by temporarily placed items, the preconditions are not met. If `control_device` queries and finds that the device is offline, an exception is also triggered. After the primitive is executed, it checks whether its return value or post-processing effect is consistent with expectations. For example, after moving to the correct position, it compares the actual coordinates with the target coordinates to see if they are within the tolerance range. After controlling the device, it queries whether the actual device state has changed successfully. If the preconditions do not match or the post-processing effect is inconsistent, the primitive is determined to have failed to execute, triggering an execution interruption. The system combines the sequence of function primitives that have not yet been executed after the current execution interruption point, the accumulated state effects of the currently executed primitives (such as partial movement completed, certain devices turned on), and the latest environmental perception information (such as updated GPS coordinates, obstacles detected by vision) into a complete replanning prompt text. The replanning prompt text is input into the large language model. Based on the current interruption state and the original target task, the model re-infers and generates a function call sequence adapted to the current scenario. This sequence may be a modification of the remaining primitives (such as a new path to bypass obstacles) or a new solution that completely replaces the original subsequent steps (such as calling the user or placing an item at the door when unable to enter the bedroom), and ultimately makes the task converge toward the original target.

[0038] In step S4, the process of the large language model regenerating a function call sequence adapted to the current state includes: When a large language model needs to converge from its current interrupted state to the target task, there may be multiple paths for its candidate function primitives. To enable the model to select the best one, the system introduces the concept of task reconstruction priority. A comprehensive score is calculated for each candidate function primitive as the basis for the selection order. This priority is obtained by weighted summation of multiple evaluation indicators. The definitions and acquisition methods of each indicator are as follows: The progress gain of the environment state relative to the target state after executing the candidate function primitive measures how close the system environment state will be to the final target task if the primitive is executed. For example, if the current state is "living room on the first floor" and the target task is "turn on the air conditioner in the bedroom on the second floor", the candidate primitive "move to the elevator" has a higher progress gain than "voice broadcast to ask the user" because the former makes the spatial location converge toward the target floor. The progress gain can be quantified by calculating the reduction of the difference between the current state and the target state in key dimensions (location, device status, etc.) or implicitly evaluated by the large language model based on semantic common sense. The semantic dependency strength between candidate function primitives and target tasks reflects the degree to which the primitive is indispensable to achieving the final goal. For example, for the goal of "turn on the air conditioner", the semantic dependency strength of "controlling the device (air conditioner)" is much higher than that of "voice broadcasting the temperature". This strength can be obtained through the weight of the associated edges between nodes in the scene knowledge graph or a predefined dependency table. The estimated time or energy cost of executing candidate function primitives is the reciprocal of the time or energy cost. This indicator is used to balance efficiency and resource consumption. The estimated time can be estimated through historical execution statistics or path planning algorithms. The estimated energy cost can be calculated by combining the device power and the estimated time, and the reciprocal of the result is used as the indicator. This gives higher weight to primitives with shorter execution time and lower energy consumption, which is in line with the efficiency-first scheduling principle.

[0039] After normalization, the three indicators mentioned above are used to obtain the task refactoring priority through a weighted summation. The weight coefficients of each indicator can be adaptively adjusted according to the current scenario, for example: When the task is urgent (such as when the user explicitly requests "as soon as possible"), the weight of the cost reciprocal metric can be increased, and primitives that are executed quickly can be selected first. When the robot's battery is low, the weight of the inverse cost metric can be increased to prioritize energy saving. When the task logic is complex and error-prone, the weights of progress gain and semantic dependency strength metrics can be increased to ensure that the goal is steadily advanced at each step. Adaptive adjustment of weight coefficients can be achieved through a rule engine (such as a predefined scene template) or a meta-learning model; The large language model selects candidate function primitives in descending order of task reconstruction priority and adds them to the new sequence. During the selection process, it continuously verifies whether the selected primitive set satisfies all the preconditions and post-effect constraints of the target task. Once the constraints are satisfied, the selection stops and the final supplementary or corrected function call sequence is output. For example, suppose a robot is performing the task of "delivering medicine to the bedroom and reminding the user to take it" but cannot enter because the bedroom door is closed. Candidate primitives include: voice prompt "remind the user to open the door", waiting in place, moving to the living room to wait, and controlling the smart lock (if supported). Through priority evaluation, if the lock is controllable and secure, controlling the lock is selected first because of its high progress gain and strong dependency. If the lock is uncontrollable, voice prompt may be better than simply waiting because of its low cost and ability to promote interaction.

[0040] Through the above technical solution, this embodiment provides a dynamic task replanning method for embodied intelligent agents. The method achieves real-time detection and interruption of execution anomalies by defining the preconditions and postconditions of the execution state machine and function primitives. Dynamic task reconstruction is achieved by concatenating the interruption context into a replanning prompt and inputting it into a large language model. By introducing a task reconstruction priority evaluation mechanism (combining three dimensions of progress gain, semantic dependency strength, and execution cost and applying adaptive weighting) to prioritize candidate function primitives until the target task constraints are met, the method solves the problems of rigid execution logic and inability to dynamically respond to environmental changes in the prior art. This enables embodied intelligent agents to autonomously generate new execution sequences adapted to the current state when encountering abnormal scenarios such as path blocking, device offline, or user changes of instructions midway. This significantly improves the robustness, adaptability, and resource utilization efficiency of task execution. At the same time, the priority evaluation mechanism ensures that the reconstruction scheme achieves a dynamic balance between logical correctness, semantic rationality, and execution economy.

[0041] In one embodiment, the method further includes: a multi-instruction concurrent scheduling and resource conflict resolution mechanism; When there are multiple composite instructions to be executed within the same scheduling cycle, the function call sequences corresponding to each instruction are merged into a global task queue; The system predefines the occupation modes (exclusive or shared) and expected occupation duration of three key resources: mobile channels, IoT devices, and computing resources. Based on this, a resource conflict detection matrix is ​​established. The rows and columns of the matrix are each function primitive and each resource item, respectively. The cells indicate the occupation requirements of the primitive for a specific resource. When multiple primitives simultaneously request the same exclusive resource, it is determined to be a resource conflict. The conflict resolution strategy is decided based on the following factors: instruction priority, which can be preset by the user or automatically inferred based on the instruction content (such as automatically increasing the priority of medical assistance), with a value range of 1 to 10; user weight, which is set based on user role or historical interaction frequency, with a value range of 0 to 1; and energy consumption constraint, which is quantified based on the ratio of the task's estimated energy consumption to the robot's current remaining power. Based on a weighted comparison of the above factors, the system reorders conflicting primitives for sequential execution or staggered batch execution, ultimately generating a parallel or sequential execution plan without resource conflicts.

[0042] The method also includes: a multi-agent cooperative extension mechanism; The function call sequence generated by the large language model is decomposed into several sub-tasks based on spatial decoupling (e.g., involving different regions), device independence (e.g., controlling multiple unrelated devices), and functional dependencies (e.g., devices can only be operated after being moved). Each subtask is assigned to a different embodied intelligent agent execution terminal, and the allocation strategy takes into account the terminal hardware capabilities, current location and current load. Establish a lightweight communication protocol (such as MQTT) between the central coordinator and each execution terminal. The central coordinator maintains a global task progress table (recording the status of each subtask, the assigned terminal, and the progress) and a shared environmental semantic map (integrating local perception information uploaded by each terminal). When any execution terminal senses a change in the environment or an execution anomaly, it uploads the updated local state to the central coordinator. The coordinator assesses the scope of the impact. If it only involves a single subtask, it triggers a local replanning (a new solution is generated autonomously by the terminal or with the assistance of the coordinator). If it affects the overall progress, it triggers a global replanning (the unfinished subtasks and new environment information are re-input into the large language model). If the terminal fails, the terminal's task is migrated to another available terminal.

[0043] Through the above technical solution, this embodiment provides a multi-task scheduling and multi-machine collaboration method for embodied intelligent agents. The method solves the resource contention problem when multiple instructions are concurrent by establishing a global task queue and a resource conflict detection matrix, and resolving conflicts by combining instruction priority, user weight and energy consumption constraints. By decomposing the single-machine task sequence into sub-tasks and distributing them to multiple terminals, establishing a central coordinator and a lightweight communication protocol to maintain global progress and a shared map, and triggering dynamic replanning and task migration in case of anomalies, multi-machine distributed collaborative execution is realized. This overcomes the defects of disordered concurrent scheduling and limited single-machine execution capability in the prior art, and significantly improves task throughput, resource utilization efficiency and system robustness.

[0044] This application also discloses a multimodal task understanding and collaborative planning system for embodied intelligent agents, characterized in that it is used to implement the multimodal task understanding and collaborative planning method for embodied intelligent agents described in any of the above claims, the system comprising: The multimodal perception module, deployed on the embodied intelligent agent platform, includes a GPS receiving unit, a visual acquisition unit, and a multimodal information fusion unit. It is used to acquire spatial positioning data and visual sensor data in real time, identify semantic features in the environment through a target detection model, and fuse the spatial positioning data and visual semantic features into a structured text description. The function primitive library module is used to store predefined standardized interface function primitives, which cover four types of capabilities: motion control, device control, voice interaction and environmental awareness. The function primitives are registered to the function toolbox for use. The large language model decision module is deployed on a cloud server or a local edge computing node. It has a built-in function call interface adaptation layer, which is used to receive the structured text description and user natural language instructions. It selects function primitives from the function toolbox through the reasoning mechanism constrained by the scene knowledge graph, autonomously reasones the task dependencies, and outputs the function call sequence composed of the selected function primitives ordered by the execution logic. The task scheduling and execution module includes a function call parser, an execution state machine, and a replanning trigger. It is used to parse the function call sequence, call the corresponding function primitives in sequence, monitor environmental state changes and execution result feedback in real time, and trigger a replanning request when execution failure is detected or the environmental state does not meet the preset preconditions. The collaborative communication module is used to establish a lightweight communication protocol between the central coordinator and each execution terminal in a multi-agent deployment scenario, so as to realize global task progress synchronization, maintenance of shared environmental semantic map, and status feedback and task scheduling of execution terminals.

[0045] Through the above technical solution, this embodiment provides a multimodal task understanding and collaborative planning system for embodied intelligent agents. The system integrates GPS spatial positioning and visual semantic features through a multimodal perception module and transforms them into structured text descriptions. A function primitive library module encapsulates four types of capabilities—motion control, device control, voice interaction, and environmental perception—into standardized interface function primitives and registers them in a function toolbox. A large language model decision module receives structured text and user instructions and generates function call sequences based on scene knowledge graph constraints. A task scheduling and execution module parses the sequences, calls the primitives, monitors the execution status, and triggers replanning in case of anomalies. A collaborative communication module enables central coordination and terminal collaboration in multi-agent deployment scenarios. This system constructs a complete closed loop from environmental perception, semantic understanding, task decision-making to execution feedback, unifying heterogeneous execution capabilities into a function call framework. It achieves dynamic decomposition and adaptive execution of composite instructions, solving the problems of semantic understanding gaps, rigid execution logic, and inconsistent interface calls in existing technologies. This significantly improves the task success rate and system scalability of embodied intelligent agents in complex dynamic environments.

[0046] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for multimodal task understanding and collaborative planning for embodied intelligent agents, characterized in that, The method includes: S1. Real-time acquisition of GPS spatial positioning data and visual sensor data of the embodied intelligent agent; identification of semantic features in the environment through a target detection model; mapping of the GPS spatial positioning data to preset spatial hierarchical semantic labels; mapping of the two-dimensional bounding box obtained by target detection of the image collected by the visual sensor to the three-dimensional spatial position in the robot coordinate system through a camera imaging geometric model; construction of a spatial semantic dynamic mapping table with spatial hierarchical semantic labels and visual object categories as joint indexes; dynamic association of physical coordinates; calculation of the current optimal mapping value based on visual detection confidence and historical access frequency; and conversion of the fusion result into a structured text description. S2. The motion control capability, IoT device control capability, and human-computer interaction feedback capability of the embodied intelligent agent are encapsulated into function primitives with standardized input and output interfaces, and a function toolbox that can be directly called by large language models is constructed. S3. Pre-construct an environmental scene knowledge graph. The nodes of the knowledge graph represent entity objects or regions in the environment, and the edges represent spatial inclusion relationships, functional association relationships, or operational dependency relationships between entities. The structured text description and the user's natural language instructions are concatenated to form a prompt input to the large language model. The large language model selects at least one function primitive from the function toolbox through its function call mechanism, and uses the subgraph structure of the knowledge graph as reasoning constraints. Based on the current environmental context and the instruction intent, it autonomously reasones the task dependency relationship and outputs a function call sequence composed of the selected function primitives ordered according to the execution logic. S4. Parse the function call sequence, and the scheduling execution engine calls the corresponding function primitives in sequence. During the function execution, continuously monitor the changes in the environment state and the feedback of the execution results. When an execution failure is detected or the environment state does not match the preset preconditions, real-time replanning is triggered, and the large language model regenerates the function call sequence adapted to the current state.

2. The multimodal task understanding and cooperative planning method for embodied intelligent agents according to claim 1, characterized in that, Step S1, the process of fusing spatial positioning data with visual semantic features and transforming it into a structured text description, includes: Map GPS spatial positioning data to preset spatial hierarchical semantic labels; The image acquired by the vision sensor is input into a lightweight object detection network, which outputs the object categories in the environment and their two-dimensional bounding boxes. Based on the camera imaging geometric model, the two-dimensional bounding box is mapped to its three-dimensional spatial position in the robot coordinate system; Construct a spatial semantic dynamic mapping table, which uses spatial hierarchical semantic labels and visual object categories as a joint index to dynamically associate the corresponding physical coordinates or relative positions; When the same semantic label corresponds to multiple physical coordinates, the current optimal mapping value is calculated based on the visual detection confidence and historical access frequency.

3. The multimodal task understanding and collaborative planning method for embodied intelligent agents according to claim 2, characterized in that, The process of calculating the current optimal mapping value includes: Define semantic tags The corresponding set of candidate physical coordinates is ,in, , For physical coordinate vectors, For visual detection confidence, For timestamps; Then the current optimal mapping coordinates Determined by the following formula: in, For the current time, The time decay constant, This is a normalization function for the frequency of historical visits. , , Preset weighting coefficients; When set When the coordinates are empty or the confidence of all candidate coordinates is below a preset threshold, an active visual search primitive is triggered to supplement the mapping data.

4. The multimodal task understanding and cooperative planning method for embodied intelligent agents according to claim 3, characterized in that, In step S2, the function primitives include at least the following categories: The location movement primitive takes target semantic points, navigation strategy patterns or path planning constraints as input parameters, calls the underlying navigation stack and returns the execution status and current position feedback. Device control primitives, whose input parameters include device type, device identifier, control action and action parameters, send instructions directly to smart home gateways or devices via IoT protocols; Voice interaction primitives, whose input parameters include the text to be read, speech rate or timbre, call the speech synthesis engine to convert the text into an audio stream and play it through the speaker; Environmental perception primitives take the region of interest or target object category as input parameters and output a Boolean value or metric value of environmental state as output. The standardized interfaces of the function primitives are defined using a unified description language and registered to the tool call interface layer of the large language model.

5. The multimodal task understanding and cooperative planning method for embodied intelligent agents according to claim 4, characterized in that, The process of generating the function call sequence is constrained and guided by the scene graph. An environmental scene knowledge graph is pre-constructed, where nodes represent entity objects or regions in the environment, and edges represent spatial inclusion relationships, functional association relationships, or operational dependencies between entities. The knowledge graph is generated through manual annotation, import of building information models, or machine autonomous exploration and mapping, and supports dynamic updates; The structured text description and user instructions are mapped to a subset of nodes in the knowledge graph, thus decomposing the task into a path search problem on the knowledge graph. The large language model uses the subgraph structure of the knowledge graph as reasoning constraints to generate a function call sequence that conforms to spatial logic and functional logic.

6. The multimodal task understanding and cooperative planning method for embodied intelligent agents according to claim 5, characterized in that, In step S4, the process of triggering real-time replanning includes: Define an execution state machine, including four states: waiting to execute, executing, executing successfully, and executing unsuccessfully. Establish a set of preconditions and a set of postconditions for function primitives. The preconditions and postconditions are described in predicate logic or key-value pair form and are consistent with the environment state representation model. When a function primitive is executed, if the precondition does not match the current environment state, or the execution result is inconsistent with the postcondition, the execution will be interrupted. The sequence of function primitives that have not been executed after the breakpoint, the cumulative effect of the currently executed primitives, and the latest environmental awareness information are concatenated into a replanning prompt, which is then input into the large language model. The large language model generates a sequence of supplementary or corrective function calls that converges from the current interrupted state to the target task.

7. The multimodal task understanding and cooperative planning method for embodied intelligent agents according to claim 6, characterized in that, In step S4, the process of the large language model regenerating a function call sequence adapted to the current state includes: Candidate function primitives are sorted according to task refactoring priority; The task reconstruction priority is obtained by weighted summation of multiple evaluation indicators; The evaluation metrics include: the progress gain of the environment state relative to the target state after executing the candidate function primitive, the semantic dependency strength between the candidate function primitive and the target task, and the reciprocal of the estimated time or energy cost of executing the candidate function primitive. The weighting coefficients used in the weighted summation are adaptively adjusted according to the current scenario, including increasing the weight of the reciprocal cost index when the task is urgent, and increasing the weight of the reciprocal cost index to prioritize energy saving when the robot's battery is low. Candidate function primitives are selected in descending order of task reconstruction priority until all preconditions and post-effect constraints of the target task are met.

8. The multimodal task understanding and cooperative planning method for embodied intelligent agents according to claim 7, characterized in that, The method further includes: When there are multiple compound instructions to be executed within the same scheduling period, the function call sequences corresponding to each compound instruction are merged into a global task queue. Based on the usage requirements of each function primitive for mobile channels, device resources, and computing resources, a resource conflict detection matrix is ​​established. The rows and columns of the matrix are each function primitive and each resource item, respectively, and the cells mark the usage requirements of the primitive for specific resources. For function primitives with resource conflicts, conflict resolution is performed based on instruction priority, user weight, or energy consumption constraints. The instruction priority ranges from 1 to 10, the user weight ranges from 0 to 1, and the energy consumption constraint is quantified based on the ratio of the task's estimated energy consumption to the robot's current remaining power, generating conflict-free parallel or serial execution plans.

9. The multimodal task understanding and cooperative planning method for embodied intelligent agents according to claim 8, characterized in that, The method further includes: The function call sequence generated by the large language model is decomposed into multiple subtasks that can be executed in parallel or collaboratively based on spatial decoupling, device independence and functional dependencies. Each subtask is assigned to a different embodied intelligent agent execution terminal, and the allocation strategy takes into account the terminal hardware capabilities, current location and current load. Establish a lightweight communication protocol between the central coordinator and each execution terminal. The central coordinator maintains a global task schedule and a shared environment semantic map. When any execution terminal senses an environmental change or execution anomaly, it uploads the updated local state to the central coordinator. The central coordinator then triggers a global replanning or a local task migration. If it only involves a single subtask, a local replanning is triggered. If it affects the overall progress, a global replanning is triggered. If a terminal fails, the terminal's task is migrated to another available terminal.

10. A multimodal task understanding and collaborative planning system for embodied intelligent agents, characterized in that, The system is used to implement the multimodal task understanding and cooperative planning method for embodied intelligent agents as described in any one of claims 1-9, the system comprising: The multimodal perception module, deployed on the embodied intelligent agent platform, includes a GPS receiving unit, a visual acquisition unit, and a multimodal information fusion unit. It is used to acquire spatial positioning data and visual sensor data in real time, identify semantic features in the environment through a target detection model, and fuse the spatial positioning data and visual semantic features into a structured text description. The function primitive library module is used to store predefined standardized interface function primitives, which cover four types of capabilities: motion control, device control, voice interaction and environmental awareness. The function primitives are registered to the function toolbox for use. The large language model decision module is deployed on a cloud server or a local edge computing node. It has a built-in function call interface adaptation layer, which is used to receive the structured text description and user natural language instructions. It selects function primitives from the function toolbox through the reasoning mechanism constrained by the scene knowledge graph, autonomously reasones the task dependencies, and outputs the function call sequence composed of the selected function primitives ordered by the execution logic. The task scheduling and execution module includes a function call parser, an execution state machine, and a replanning trigger. It is used to parse the function call sequence, call the corresponding function primitives in sequence, monitor environmental state changes and execution result feedback in real time, and trigger a replanning request when execution failure is detected or the environmental state does not meet the preset preconditions. The collaborative communication module is used to establish a lightweight communication protocol between the central coordinator and each execution terminal in a multi-agent deployment scenario, so as to realize global task progress synchronization, maintenance of shared environmental semantic map, and status feedback and task scheduling of execution terminals.

Images