Embodied robot control method and system based on multi-segment task planning

Abstract

The application relates to a somatotype robot control method and system based on a multi-section task planning. The method comprises the following steps: acquiring input information; determining a structured task graph based on a user instruction and current observation information through a task planning layer in a task execution model; determining a current task node from the structured task graph, and determining an intermediate target representation based on the current task node, the current observation information and robot state data through a preset world model layer in the task execution model; determining a short-time action block for a somatotype robot to execute the current task node based on the current observation information, the robot state data and the intermediate target representation through a somatotype execution layer in the task execution model based on a OneModel architecture; executing an action corresponding to the short-time action block, and updating task execution data of the somatotype robot according to a matching condition of an execution result of the short-time action block and the intermediate target representation.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to an embodied robot control method and system based on multi-segment task planning. Background Technology

[0002] Vision-Language-Action (VLA) models typically generate actions directly from the embodied robot's current observations and user commands, i.e., "Observation + Instruction → Action". This approach is suitable for short, single-stage tasks, but in real-world robotic tasks, many operations require breaking down the task, determining the target state after each step, and finally executing the action.

[0003] Existing world models are often understood as prediction models of future videos or scenes, and their outputs may not be directly applicable to robot control. Existing VLA models also often lack explicit intermediate goals and success criteria, making it difficult to determine whether a robot failure is due to incorrect task breakdown, incorrect target state, or incorrect action execution. Summary of the Invention

[0004] In view of the above, this application provides a method and system for controlling an embodied robot based on multi-segment task planning, the purpose of which is to solve the above-mentioned technical problems.

[0005] In a first aspect, this application provides a control method for an embodied robot based on multi-segment task planning, the method comprising: Acquire input information; the input information includes user commands, current observation information, and robot state data of the embodied robot; The task planning layer in the task execution model determines a structured task graph based on the user instructions and the current observation information; the structured task graph includes multiple task nodes with dependencies required to complete the user instructions; The current task node is determined from the structured task graph, and based on the current task node, the current observation information, and the robot state data, an intermediate target representation is determined through a preset world model layer in the task execution model; the intermediate target representation is used to represent the target state that should be reached after the current task node is completed. Based on the current observation information, the robot state data, and the intermediate target representation, the embodied execution layer in the task execution model determines a short-term action block for the embodied robot to execute the current task node. The action corresponding to the short-time action block is executed, and the task execution data of the embodied robot is updated according to the matching between the execution result of the short-time action block and the intermediate target representation.

[0006] Secondly, this application provides a hymened robot control system based on multi-segment task planning, the hymened robot control system based on multi-segment task planning includes: The acquisition module is used to acquire input information, including user commands, current observation information, and robot state data of the embodied robot. The first determining module is used to determine a structured task graph based on the user instructions and the current observation information through the task planning layer in the task execution model; the structured task graph includes multiple task nodes with dependencies required to complete the user instructions; The second determining module is used to determine the current task node from the structured task graph, and based on the current task node, the current observation information, and the robot state data, determine the intermediate target representation through the preset world model layer in the task execution model; the intermediate target representation is used to represent the target state that should be reached after the current task node is completed. The third determining module is used to determine, through the embodied execution layer in the task execution model, a short-term action block for the embodied robot to execute the current task node, based on the current observation information, the robot state data, and the intermediate target representation. The execution module is used to execute the actions corresponding to the short-time action block, and update the task execution data of the embodied robot according to the matching of the execution result of the short-time action block with the intermediate target representation.

[0007] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the embodied robot control method based on multi-segment task planning as described in any embodiment of the first aspect.

[0008] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the embodied robot control method based on multi-segment task planning as described in any embodiment of the first aspect.

[0009] The technical solutions provided in this application have the following advantages compared with the prior art: (1) By explicitly breaking down long-term complex tasks into a three-stage hierarchical structure of task graph, intermediate target and short-term action block, the robot can clearly understand the target state of each execution stage and have clear criteria for success judgment; (2) With the help of the closed-loop matching mechanism between intermediate target representation and actual execution result, the source of deviation can be accurately determined after the action fails, effectively avoiding the error accumulation and fault location problems caused by end-to-end direct mapping, and significantly improving the control stability and execution interpretability of multi-stage embodied tasks. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a preferred embodiment of the embodied robot control method based on multi-segment task planning in this application; Figure 2 This is an exemplary flowchart illustrating the determination of a structured task graph according to some embodiments of this application; Figure 3 This is an exemplary flowchart illustrating the determination of intermediate target representations according to some embodiments of this application; Figure 4 This is an exemplary flowchart illustrating the determination of short-time action blocks according to some embodiments of this application; Figure 5 This is an exemplary flowchart illustrating the determination of an end effector pose increment sequence according to some embodiments of this application; Figure 6 This is an exemplary flowchart illustrating the updating of task execution data according to some embodiments of this application; Figure 7 This is an exemplary flowchart of hierarchical replanning based on deviation values, as shown in other embodiments of this application; Figure 8 This is an exemplary flowchart of updating the structured task graph according to other embodiments of this application; Figure 9 This is an exemplary flowchart illustrating the updating of short-time action blocks according to some embodiments of this application; Figure 10 This is an exemplary schematic diagram of the model structure of the task execution model shown in some embodiments of this application; Figure 11 This is a schematic diagram of a preferred embodiment of the embodied robot control system based on multi-segment task planning in this application; Figure 12 This is a schematic diagram of a preferred embodiment of the electronic device of this application; The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0014] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0015] Reference Figure 1 The diagram shown is a flowchart illustrating an embodiment of the embodied robot control method based on multi-segment task planning according to this application. The method is executed by an electronic device (such as a processor), which can be implemented by a software system and / or a hardware system. This embodied robot control method based on multi-segment task planning includes: Step 101: Obtain input information; the input information includes user instructions, current observation information, and robot state data of the embodied robot.

[0016] Input information refers to the data used to drive the task execution model to perform multi-stage task planning.

[0017] User instructions are descriptions of tasks to be performed issued by users in the form of natural language or structured commands. Examples include folding the T-shirts on the table or putting the cup in the sink.

[0018] Current observation information refers to the multimodal raw data acquired by the embodied robot at the current moment through its various onboard sensors as it perceives its physical environment. For example, current observation information may include current scene images captured by a camera, scene depth maps acquired by a depth camera, etc. In some embodiments, current observation information includes visual images of clothes to be folded on a folding table.

[0019] Robot state data refers to a set of parameters used to characterize the physical structure and pose of an embodied robot, including the joint angles of the robotic arm, the pose of the end effector, the opening and closing state of the gripper, the pose of the mobile chassis, and the reachable space range.

[0020] In some embodiments, the processor acquires input information by: synchronously acquiring current scene images and depth scene maps through multimodal sensors as current observation information; reading the encoder values ​​of each joint and the pose matrix of the end effector in the base coordinate system from the embodied robot (hereinafter also referred to as the robot) controller as robot state data; and receiving text instructions input by the user through a natural language interface.

[0021] In some embodiments, the processor can also receive user voice commands via a voice recognition module and convert them into text commands.

[0022] Step 102: Determine a structured task graph based on the user instructions and the current observation information through the task planning layer in the task execution model.

[0023] A structured task graph is a graph-structured data structure formed by breaking down a complex task corresponding to a user instruction into multiple ordered subtasks. Each node in a structured task graph represents an independently executable and verifiable subtask, and the directed edges between nodes represent the logical temporal dependencies between the subtasks. A structured task graph includes multiple dependent task nodes required to complete a user instruction.

[0024] Task nodes are the basic building blocks of a structured task graph, with each node corresponding to a subtask. A subtask includes an execution objective, execution prerequisites, and execution completion conditions. Dependencies refer to the pre- and post-execution constraints between different task nodes. For example, the "grab the cup" node must be executed after the "approach the cup" node, and the "place in the sink" node must be executed after the "grab the cup" node.

[0025] In some embodiments, the processor determines the structured task graph by: performing semantic parsing of user instructions to extract task objectives and key operational objects; performing scene recognition on current observation information to determine interactive objects and their spatial locations in the scene; generating a sequence of subtasks from the initial state to the target state based on the task objectives and interactive objects, combined with capability boundary information such as the kinematic reachability and gripper load capacity of the embodied robot; and establishing directed edges based on the logical sequence relationship between the subtasks to form a structured task graph.

[0026] Further explanation on determining the structured task graph can be found below. Figure 2 And its related descriptions.

[0027] Step 103: Determine the current task node from the structured task graph, and based on the current task node, the current observation information, and the robot state data, determine the intermediate target representation through the preset world model layer in the task execution model.

[0028] The current task node refers to the graph node corresponding to the subtask currently to be executed, selected from the structured task graph.

[0029] The pre-trained world model layer is a network layer in the task execution model that is pre-trained to predict future states based on the current state and task conditions.

[0030] An intermediate target representation is a mathematical expression of the target state expected to be observed after completing the current task node. The intermediate target representation provides a basis for success determination in the generation of subsequent actions. It can be used to characterize the target state that should be achieved after completing the current task node. In some embodiments, the specific form of the intermediate target representation includes any one of the following: sub-target image, target state diagram, semantic target description, 3D target pose, keypoint heatmap, object segmentation mask, target region mask, contact point distribution, gripping point location, placement point coordinates, availability representation, or latent space feature vector.

[0031] In some embodiments, the processor determines the intermediate target representation by: reading the currently executed task node from the structured task graph and obtaining the task semantics and completion condition information of the node; inputting the image from the current observation information into a visual encoder to extract visual feature vectors, and inputting the end effector pose and joint angle from the robot state data into a pose encoder to extract pose feature vectors; inputting the visual feature vector, pose feature vector, and task embedding vector of the task semantics into a preset world model layer; the world model layer generates a target state distribution that should be reached after the current task node is completed based on the above inputs, and samples the intermediate target representation from the distribution.

[0032] Further explanation regarding the determination of intermediate target representations can be found below. Figure 3And its related descriptions.

[0033] Step 104: Through the embodied execution layer in the task execution model, based on the current observation information, the robot state data, and the intermediate target representation, determine the short-term action block for the embodied robot to execute the current task node.

[0034] A short-time action block is a sequence of motion commands composed of multiple consecutive frames of motion parameters arranged in chronological order. Short-time action blocks can be used to drive an embodied robot to complete a motion process toward a central target representation within a short time window. In some embodiments, each frame of motion parameters in a short-time action block includes at least one of the following: end effector pose increment, end effector target pose, target values ​​for each joint angle, angular velocity commands for each joint, gripper opening and closing control commands, linear and angular velocities of the moving chassis, or multi-robot cooperative motion commands.

[0035] In some embodiments, the processor determines short-term action blocks by: extracting a current state representation from current observation information; calculating the feature space difference between the current state representation and the intermediate target representation to obtain state difference features; concatenating the state difference features with the current end effector pose and the semantic description of the current task node in the robot state data to form policy input features; inputting the policy input features into a pre-trained low-level action policy network, which adopts a diffusion strategy or a generative architecture based on action block transformation, and outputs a sequence of continuous multi-frame action parameters covering dozens of future control time steps; and combining the sequence into short-term action blocks in chronological order.

[0036] For further explanation on determining short-time action blocks, please see the following text. Figure 4 And its related descriptions.

[0037] Step 105: Execute the action corresponding to the short-time action block, and update the task execution data of the embodied robot according to the matching of the execution result of the short-time action block with the intermediate target representation.

[0038] Task execution data refers to the set of information used to record and track the execution status of each task node during the execution of a structured task graph by an embodied robot. Task execution data includes the robot's state data, the execution status of the current task node, the execution result, execution feedback information, and / or historical execution information.

[0039] The matching status between the execution result and the intermediate target representation refers to the judgment result obtained by comparing the degree of consistency between the actual environmental observation after execution and the preset intermediate target representation.

[0040] In some embodiments, the processor executing actions and updating task execution data includes: converting the action parameters of each frame in the short-time action block into joint space control commands after inverse kinematics solving and trajectory smoothing, and sequentially sending them to the robot servo driver for execution; after execution, collecting actual observation information through sensors; encoding the actual observation information into an actual observation representation, and calculating the cosine similarity or structural similarity index between the actual observation representation and the intermediate target representation as a matching score; when the matching score is higher than a preset completion threshold, determining that the current task node has been successfully executed, updating the node execution status of the task node in the task execution data to the successful execution status; when the matching score is lower than the preset completion threshold, determining that the current task node has failed, updating the node execution status to the failed execution status, and recording the failure type as action layer deviation.

[0041] The processor can execute the actions corresponding to the short-term action blocks and update the task execution data of the embodied robot based on the matching of the execution results of the short-term action blocks with the intermediate target representation. Based on the updated task execution data and the dependencies between task nodes in the structured task graph, the processor determines the next task node until the task completion conditions corresponding to the user instruction are met.

[0042] Specifically, after executing the action corresponding to the short-time action block, the task execution data can be updated based on the matching between the execution result of the short-time action block and the intermediate target representation. The task execution data can include at least one of the following: the execution status of the current task node, execution success flag, execution failure flag, execution progress, exception type, set of completed task nodes, set of task nodes to be executed, robot pose change information, and environmental state change information. The task execution model can determine the next task node based on the updated task execution data and the dependencies between task nodes in the structured task graph. When the execution result of the current task node satisfies the intermediate target representation, the current task node can be marked as completed, and task nodes that depend on the current task node and whose preceding dependent tasks have all been completed can be determined from the structured task graph as the next task node. When multiple candidate task nodes satisfy the execution conditions, the next task node can be determined based on at least one of the following: priority of the candidate task node, task cost, robot current state, current position, and environmental reachability. When the execution result of the current task node does not meet the intermediate target representation, the task execution data can be updated based on the deviation between the execution result and the intermediate target representation. Based on the updated task execution data, the intermediate target representation corresponding to the current task node can be re-determined, or a short-term action block for executing the current task node can be re-determined. If the current task node meets the preset failure conditions, an alternative task node, a rollback task node, or a re-planned task node can be determined based on the dependencies in the structured task graph. The robot can schedule task nodes in the structured task graph sequentially or in parallel according to a closed-loop method of "determining the current task node—determining the intermediate target representation—generating a short-term action block—executing the action—updating the task execution data—determining the next task node" until the task completion conditions corresponding to the user instruction are met. Task completion conditions may include the completion of all target task nodes in the structured task graph, the confirmation of the target state corresponding to the user instruction by the current observation information, or the task execution data indicating that the user instruction has been completed.

[0043] Further instructions regarding updating task execution data can be found below. Figure 6 And its related descriptions.

[0044] In some embodiments of this application, by explicitly decomposing long-term complex tasks into a three-stage hierarchical structure of task graph, intermediate target, and short-term action block, the robot can clearly understand the target state of each execution stage and have clear criteria for success determination. With the help of the closed-loop matching mechanism between intermediate target representation and actual execution result, the source of deviation can be accurately diagnosed after action failure, effectively avoiding the error accumulation and fault location problems caused by end-to-end direct mapping, and significantly improving the control stability and execution interpretability of multi-stage embodied tasks.

[0045] Figure 2 This is an exemplary flowchart illustrating the determination of a structured task graph according to some embodiments of this application. In some embodiments, Figure 2 The process shown can be executed by an electronic device, such as a processor.

[0046] Step 201: Perform semantic parsing processing based on the user instructions to obtain the task target.

[0047] A task objective is a set of elements extracted from user instructions and described in a structured manner, representing the operation to be performed. A task objective includes the target action type and the target object. Target action types include stacking, grasping, moving, placing, or folding; target objects include items such as a T-shirt, a cup, or a sink.

[0048] In some embodiments, the processor obtains the task objective by: inputting user instructions into a pre-trained large language model for semantic encoding; setting an action type classification head and an operation object extraction head at the decoding end of the large language model, and outputting action type labels and operation object noun phrases respectively; and combining the action type labels and operation object noun phrases into a structured task objective.

[0049] In some embodiments, the processor can also obtain the task target by combining keyword matching and syntactic analysis. For example, it can obtain the target action type by matching a preset action lexicon from the user instructions, and then extract the object component on which the action type depends as the target operation object through syntactic analysis to complete the extraction of the task target.

[0050] Step 202: Based on the current observation information, perform scene recognition processing to determine the operation object information and scene constraint information.

[0051] Operational object information refers to the set of perceptual attributes associated with the target operational objects involved in the task objective within the current observation information. Operational object information includes at least one of the following: category label of the operational object, two-dimensional bounding box, three-dimensional bounding box, instance segmentation mask, three-dimensional point cloud cluster, and pose information.

[0052] Scene constraint information refers to the set of environmental constraints in the current observation information that affect the robot's actions. Scene constraint information includes the spatial positional relationship between the manipulated object and surrounding objects, support surface information, obstacle distribution, passable areas, and workspace boundaries.

[0053] In some embodiments, the processor determines the operation object information and scene constraint information by: inputting the image from the current observation information into the target detection network and outputting the two-dimensional bounding box and category label of the operation object; fusing the depth map or point cloud data from the current observation information with the two-dimensional bounding box and calculating the three-dimensional bounding box and center pose of the operation object in the camera coordinate system as the operation object information; and simultaneously performing panoramic segmentation and spatial layout analysis on the image and depth map to extract the relative distances between objects, occlusion relationships, and support surface normals as scene constraint information.

[0054] Step 203: Based on the task objective, the operation object information, the scene constraint information, and the robot capability boundary information associated with the embodied robot, determine the multiple sub-tasks required to complete the task objective and the logical temporal relationship between the multiple sub-tasks.

[0055] Robot capability boundary information is a set of prior parameters used to limit the range of motion that an embodied robot can perform. Robot capability boundary information includes the range of motion limits of each joint of the robotic arm, maximum linear velocity and angular velocity, maximum load capacity, gripping force range of the end effector, and the reachable workspace of the base.

[0056] Multiple subtasks refer to multiple concrete operational units that need to be executed sequentially to achieve a task objective. Logical temporal relationships refer to the dependencies between subtasks in their execution order, including sequential execution relationships and conditional execution relationships. A sequential execution relationship means that one subtask can only begin after another subtask has been completed; a conditional execution relationship means that the execution of one subtask depends on the success or failure of the previous subtask.

[0057] In some embodiments, the processor retrieves matching candidate operation primitive sequences from a predefined operation primitive library based on the target action type in the task objective; performs spatial reachability verification on each candidate operation primitive sequence based on the pose and size of the operation object in the operation object information and the support surface position and obstacle distribution in the scene constraint information; eliminates candidate sequences that do not meet the robot capability boundary information, and selects the one with the shortest execution path from the remaining candidate sequences as multiple subtasks; establishes the sequential dependency relationship between adjacent subtasks based on the execution order of the multiple subtasks to obtain the logical temporal relationship.

[0058] Step 204: Determine multiple task nodes corresponding to the multiple subtasks based on the multiple subtasks and the logical timing relationship.

[0059] A task node is an executable unit formed by encapsulating each subtask as a vertex in a graph structure. Task nodes carry the execution semantic information of the corresponding subtask.

[0060] In some embodiments, the processor creates a corresponding graph vertex for each subtask; according to the sequential dependency constraints in the logical temporal relationship, directed edges are added between the graph vertices corresponding to subtasks that have a sequential execution relationship; thereby forming a node relationship skeleton composed of multiple graph vertices and directed edges, where each graph vertex in the skeleton is a task node.

[0061] Step 205: Determine the node execution information corresponding to the multiple task nodes, and construct a structured task graph based on the multiple task nodes and the node execution information.

[0062] Node execution information refers to the set of metadata associated with each task node, used to guide node execution and status verification. Node execution information includes subtask name, operation object identifier, precondition information, completion condition information, and failure handling strategy.

[0063] Precondition information is used to characterize the state conditions that the embodied robot must meet before performing the action in order to execute the task node. For example, for the cup-grabbing node, the preconditions are that the gripper is in an open state and the end effector is located within a preset range near the cup handle.

[0064] Completion condition information is used to characterize the state conditions required for the embodied robot to perform actions after executing the task node. For example, for the cup-grabbing node, the completion condition is that the gripper is closed and the bottom of the cup is off the table.

[0065] In some embodiments, the processor constructs a structured task graph by: extracting the pose and size of the operation object from the operation object information based on the operation object identifier of the task target; filling the operation object identifier field for each task node; setting the pose arrival condition in the precondition information and the pose determination condition in the completion condition information based on the pre-task and post-task of each task node; setting the failure handling strategy to jump to the task graph replanning entry point when the current node fails to execute; and combining all task nodes with completed node execution information with directed edges corresponding to logical timing relationships to form a structured task graph.

[0066] In some embodiments, each task node in the structured task graph corresponds to node attribute information. The node attribute information includes subtask name, operation object identifier, prerequisite information, completion condition information, and failure handling strategy.

[0067] Node attribute information is a collection of metadata attached to each task node in the structured task graph. This collection describes all decision parameters and strategy configurations required by the task node before, during, and after execution.

[0068] Failure handling strategy refers to the actions taken by the processor when the completion conditions of a task node are not met after execution. Failure handling strategies include regenerating short-term action blocks, regenerating intermediate target representations, or returning to the task planning layer to replan the structured task graph.

[0069] In this embodiment, natural language instructions are transformed into structured task graphs through semantic parsing and scene recognition, enabling long-term complex tasks to have clear sub-task decomposition and dependency expression. The introduction of robot capability boundary information ensures the physical feasibility of the planned task at the execution level. Each task node independently carries preconditions and completion conditions, providing fine-grained judgment criteria for failure diagnosis and single-node retry in the subsequent execution process, significantly improving the interpretability and robustness of task planning.

[0070] Figure 3 This is an exemplary flowchart illustrating the determination of intermediate target representations according to some embodiments of this application. In some embodiments, Figure 3 The process shown can be executed by an electronic device, such as a processor.

[0071] Step 301: Extract visual features from the current observation information to obtain the first feature vector.

[0072] The first eigenvector is a high-dimensional numerical representation extracted from the visual data in the current observation information. The first eigenvector can be expressed as any form of global image features, multi-scale pyramid features, or dense pixel-level feature maps.

[0073] In some embodiments, the processor obtains the first feature vector by: inputting the image from the current observation information into a pre-trained visual Transformer encoder, which divides the image into multiple image blocks and performs linear projection and self-attention encoding on each image block, average pooling the feature outputs of all image blocks along the channel dimension to obtain a global visual feature vector, and using the global visual feature vector as the first feature vector.

[0074] In some embodiments, the processor can also concatenate the image and depth map in the current observation information along the channel dimension to form a four-channel input tensor, and then input the four-channel input tensor into a convolutional neural network for multi-layer convolution and downsampling processing, and extract the features after global average pooling from the last convolutional feature map of the convolutional neural network as the first feature vector.

[0075] Step 302: Extract pose features from the robot state data to obtain a second feature vector.

[0076] The second eigenvector is a numerical representation extracted from the pose-related parameters in the robot's state data. The second eigenvector is used to express the embodied robot's current body configuration and end effector spatial position.

[0077] In some embodiments, the processor obtains the second feature vector by: concatenating the joint angle values, three-dimensional position coordinates of the end effector, and three-dimensional attitude Euler angles in the robot state data into a one-dimensional original pose vector in a fixed order; inputting the one-dimensional original pose vector into a multilayer perceptron for feature transformation and dimension mapping, and outputting a fixed-dimensional deep pose feature as the second feature vector.

[0078] In some embodiments, the processor may further encode the end effector position and orientation in the robot state data into position embedding vectors and orientation embedding vectors, respectively, map each joint angle value into a joint angle embedding vector through a sinusoidal position encoding function, and then obtain a second feature vector by weighted summation of the position embedding vector, orientation embedding vector and joint angle embedding vector.

[0079] Step 303: Encode the task semantics of the current task node and the node attribute information corresponding to the current task node to obtain the task embedding vector.

[0080] The task embedding vector refers to the unified numerical representation obtained by fusing and encoding the semantic description information carried by the current task node and the condition constraint information in the node attribute information. The node attribute information includes the precondition information associated with the embodied robot. The precondition information associated with the embodied robot in the node attribute information refers to the constraints that the robot's state must meet before executing the task node. For example, the end effector must be located within a preset proximity area of ​​the manipulated object, the gripper must be in an open state, or the joint angles of the robotic arm must be within safety limits.

[0081] Task semantics refers to the semantic description of the subtasks corresponding to the current task node. Task semantics includes the subtask name and the identifier of the operation object.

[0082] In some embodiments, the processor obtains the task embedding vector by: concatenating the subtask name and operation object identifier of the current task node into a text sequence, and formatting each condition item in the precondition information into structured text; inputting the text sequence and structured text together into a text encoder for semantic encoding to obtain a semantic feature vector; and mapping the semantic feature vector to the same feature space dimension as the first feature vector and the second feature vector through a linear projection layer to obtain the task embedding vector.

[0083] Step 304: Input the first feature vector, the second feature vector, and the task embedding vector into the preset world model layer in the task execution model to obtain the predicted state distribution.

[0084] The predicted state distribution refers to the set of probability distributions formed by multiple candidate target states that can be reached after executing the current task node, as output by the world model layer, and their corresponding probability values. Each candidate state in this distribution corresponds to a possible execution outcome scenario. The predicted state distribution is used to characterize the distribution of candidate states after executing the current task node.

[0085] In some embodiments, the processor obtains the predicted state distribution by: concatenating the first feature vector and the second feature vector along the feature dimension and inputting them into the scene encoding module in the world model layer to obtain a compact implicit representation of the current scene; performing cross-attention fusion of the compact implicit representation with the task embedding vector in the conditional fusion module in the world model layer to obtain conditional latent variables; inputting the conditional latent variables into the state decoding module in the world model layer, the state decoding module outputting the mean vector and variance vector of multiple candidate states, and the predicted state distribution is composed of the mean vector and variance vector of these candidate states.

[0086] In some embodiments, inputting the first feature vector, the second feature vector, and the task embedding vector into a preset world model layer in the task execution model to obtain the predicted state distribution may include the following operations: S11, construct the spatial features of the current scene where the embodied robot is located based on the first feature vector and the second feature vector.

[0087] Spatial features are representational data used to characterize the relative relationships between the geometric structure and spatial layout of objects in the current environment of an embodied robot and the robot's own pose. Spatial features, incorporating visual appearance information and robot pose information, form a feature representation within a unified spatial coordinate framework. The spatial features of the current scene can be represented in any form, such as a feature point cloud in three-dimensional space, an occupancy feature map on a voxel mesh, or a two-dimensional spatial feature map from a bird's-eye view.

[0088] In some embodiments, the processor constructs spatial features by: extracting a two-dimensional feature map corresponding to the image from a first feature vector, and extracting the three-dimensional spatial position coordinates of the end effector from a second feature vector; mapping each feature pixel in the two-dimensional feature map onto a ray in three-dimensional space through inverse perspective projection transformation according to a pre-calibrated camera intrinsic and extrinsic parameter matrix; calculating the three-dimensional coordinates of the feature point in the camera coordinate system by combining the depth value corresponding to the pixel position in the depth map, and then concatenating the three-dimensional coordinates with the three-dimensional position coordinates of the end effector to obtain a spatial feature point cloud with pose annotation.

[0089] S12, Based on the task embedding vector, the spatial features are conditionally constrained and encoded to obtain conditional features.

[0090] Conditional constraint coding refers to the coding process that uses the task semantics and precondition information contained in the task embedding vector to perform weighted modulation or affine transformation on spatial features, so that the regions in the spatial features related to the current task are enhanced and the regions unrelated to the current task are suppressed.

[0091] Conditional features are spatial feature enhancements highly aligned with the task objective, obtained after conditional constraint encoding. These conditional features drive subsequent world model layers to generate predicted state distributions that match the task instructions.

[0092] In some embodiments, the processor performs conditional constraint encoding by: passing the task embedding vector through a conditional mapping network consisting of a fully connected layer and an activation function, and outputting a set of channel-level scaling factors and channel-level bias terms; applying the channel-level scaling factors and channel-level bias terms to each feature channel of the spatial features, performing a feature-level affine transformation on the spatial features, and using the transformed features as conditional features, wherein the values ​​of the scaling factors and bias terms are constrained by the preconditional information in the task embedding vector.

[0093] In some embodiments, the processor may also use the task embedding vector as a query vector, the feature vector of each spatial location in the spatial features as a key vector and a value vector, calculate the attention weight between the task embedding vector and each spatial location through a cross-attention mechanism, and perform a weighted summation of the feature vectors of all spatial locations in the spatial features according to the attention weights to obtain the conditional features that are depth-aligned with the task embedding vector.

[0094] S13, Based on the conditional features, a predicted state distribution is generated through the preset world model layer in the task execution model.

[0095] The predicted state distribution includes at least one of the following: predicted image distribution, predicted 3D point cloud distribution, or predicted key point thermal distribution.

[0096] The predicted image distribution refers to the probability distribution of future observed images output by the world model layer after executing the current task node, using conditional features as input. Each sample in the predicted image distribution corresponds to a predicted sub-target image. The predicted 3D point cloud distribution refers to the probability distribution of the future scene's 3D point cloud output by the world model layer after executing the current task node. Each sample in the predicted 3D point cloud distribution corresponds to a set of 3D spatial point coordinates. The predicted keypoint heatmap distribution refers to the probability distribution of the 2D or 3D spatial locations of preset keypoints in the future scene after executing the current task node, output by the world model layer. High-probability regions in the predicted keypoint heatmap distribution correspond to the target spatial locations where keypoints should be after executing the current task node; preset keypoints include the grasping point of the manipulated object, the endpoint of the folding axis, or the placement target point, etc.

[0097] In some embodiments, the processor generates a predicted state distribution by: inputting conditional features into a diffusion decoder in a preset world model layer, wherein the diffusion decoder performs an inverse denoising sampling process guided by the conditional features, and gradually recovers data samples of the target mode from standard Gaussian noise during the sampling process; when the target mode is an image, the diffusion decoder outputs multiple predicted images along with the generation confidence scores of each image, and the multiple predicted images and their confidence scores constitute a predicted image distribution; when the target mode is a three-dimensional point cloud, the diffusion decoder outputs multiple point cloud sample sets and their confidence scores, constituting a predicted three-dimensional point cloud distribution.

[0098] In this embodiment, by fusing visual features and pose features into a unified spatial feature and then encoding conditional constraints using task embedding vectors, the world model layer can simultaneously perceive information from three dimensions: scene geometry, robot configuration, and task semantic constraints when generating the predicted state distribution. As a result, the output predicted image distribution, predicted 3D point cloud distribution, or predicted key point heat distribution are all highly correlated with the current task node, effectively improving the environmental adaptability and task relevance of the intermediate target representation generation.

[0099] Step 305: Based on the precondition information and the robot state data, perform reachability assessment on the candidate states in the predicted state distribution to obtain the target state that should be reached after completing the current task node, and determine the intermediate target representation based on the target state.

[0100] Reachability assessment is the process of determining whether each candidate state in the predicted state distribution can be safely reached from the current robot state in a real physical scenario using the robot's current capabilities.

[0101] In some embodiments, the processor performs reachability assessment and determines intermediate target representation by: for each candidate state in the predicted state distribution, calculating the motion change of each joint of the robot based on the difference between the expected pose of the end effector in the candidate state and the actual pose of the end effector in the current robot state data; checking whether the motion change exceeds the maximum range of motion and maximum speed limit of each joint as defined in the robot capability boundary information; after eliminating candidate states that exceed the robot capability boundary information, selecting the candidate state with the highest distribution probability from the remaining candidate states as the target state; and determining the sub-target image or three-dimensional target pose corresponding to the target state as the intermediate target representation.

[0102] In some embodiments, the processor may also perform inverse kinematics solution and collision detection once for each candidate state in the predicted state distribution, determine the candidate state with inverse kinematics solution and collision detection pass as reachable candidate state, select the best one from all reachable candidate states according to the weighted score between the predicted probability value and the length of the end effector movement path as the target state, and then map the target state into a fixed-dimensional latent space feature vector through the encoder as an intermediate target representation.

[0103] In this embodiment, visual features, pose features, and task semantic features are extracted, fused, and then input into the world model layer. This enables the world model to predict multiple possible states after executing a task node, provided that it fully perceives the current scene and the robot's state. Then, through reachability evaluation, the target state that meets both physical constraints and has the highest probability of success is selected from the prediction distribution. This ensures that the generated intermediate target representation is both verifiable and executable, effectively solving the problem that existing world models are difficult to use directly for robot control as video prediction modules.

[0104] Figure 4 This is an exemplary flowchart illustrating the determination of short-time action blocks according to some embodiments of this application. In some embodiments, Figure 4 The process shown can be executed by an electronic device, such as a processor.

[0105] Step 401: Determine the current state representation based on the current observation information, obtain the state difference between the current state representation and the intermediate target representation, and obtain the state difference feature.

[0106] Current state representation refers to the numerical vector or tensor obtained by extracting features from the current observation information, which describes the current scene state of the embodied robot.

[0107] State difference features refer to the measurement of the difference between the current state representation and the intermediate target representation in the feature space. State difference features are used to quantify the direction and magnitude of the deviation between the current scene state and the target scene state.

[0108] In some embodiments, the processor obtains the state difference features by: inputting the image in the current observation information into the visual encoder, extracting the current visual feature vector corresponding to the current observation information as the current state representation; projecting the current state representation and the intermediate target representation into the same feature space respectively, calculating the element-wise difference between the two as the preliminary difference vector; and then normalizing the preliminary difference vector to obtain the state difference features.

[0109] Step 402: The state difference features, the current end effector pose in the robot state data, and the task semantics of the current task node are concatenated to obtain the policy input features.

[0110] The current end effector pose refers to the set of three-dimensional spatial position and three-dimensional orientation of the end effector of the embodied robot relative to the world coordinate system or base coordinate system at the current moment.

[0111] Policy input features refer to a unified feature vector formed by concatenating the state difference features, the encoding of the current end effector pose, and the encoding of the task semantics of the current task node along the feature dimension.

[0112] In some embodiments, the processor obtains policy input features by: flattening the three-dimensional position coordinates and quaternion pose representation in the current end effector pose into a one-dimensional pose vector and then mapping it into a pose embedding vector through a linear layer; encoding the subtask name of the current task node into a semantic embedding vector through a text encoder; and directly concatenating the state difference features, pose embedding vector, and semantic embedding vector along the feature dimension to obtain policy input features.

[0113] In some embodiments, the processor may also use the state difference feature as the main feature, and interact and fuse the current end effector pose encoding and task semantic encoding with the state difference feature through a cross-attention mechanism. The features after the two interaction fusions are added together and then subjected to layer normalization to obtain the policy input feature.

[0114] Step 403: Input the policy input features into a preset low-level action policy network to generate continuous multi-frame action parameters.

[0115] Low-level action policy networks refer to pre-trained deep neural network models used to map high-level semantic conditions and state differences into specific action sequences of embodied robots.

[0116] Continuous multi-frame motion parameters refer to the motion command data of each frame in a sequence of motion commands covering multiple future time steps, output at once by the low-level motion policy network. Each frame's motion parameters include at least one of the following: end effector pose increment, gripper opening / closing parameters, joint control parameters, and motion confidence.

[0117] The end effector pose increment refers to the change in position and orientation of the end effector relative to the pose of the previous frame in each frame. Gripper opening / closing parameters refer to the target opening / closing degree or opening / closing velocity command of the gripper in each frame. Joint control parameters refer to the target angle, angular velocity, or torque command of each joint in each frame. Action confidence refers to the reliability score given by the low-level motion policy network for the action parameters output in each frame.

[0118] In some embodiments, the processor generates continuous multi-frame motion parameters by: inputting policy input features into a low-level motion policy network based on a diffusion policy framework, wherein the network performs a diffusion denoising process conditioned on the policy input features, and gradually denoises from a random noise sequence to generate a complete motion sequence containing T future time steps, wherein each frame of the motion sequence contains the end effector pose increment, gripper opening and closing parameters and corresponding motion confidence, wherein T is a preset motion block length.

[0119] In some embodiments, the motion parameters for consecutive frames include an end effector pose increment sequence or a joint angle increment sequence.

[0120] The end effector pose increment sequence refers to the ordered set of consecutive frames of end effector pose increments output by the low-level action policy network, arranged in chronological order. Each element in this ordered set corresponds to the change in position and orientation of the end effector relative to the previous frame at a given time step.

[0121] The joint angle increment sequence refers to the ordered set formed by arranging the changes of each joint angle of the robotic arm relative to the previous frame in a chronological order across multiple consecutive frames, corresponding to the end effector pose increment sequence.

[0122] In some embodiments, the processor generates continuous multi-frame motion parameters by: the low-level motion policy network directly outputting an end effector pose increment sequence, each frame of which contains the translation increment and rotation increment of the end effector along the three coordinate axes in the local motion coordinate system; or the low-level motion policy network directly outputting a joint angle increment sequence, each frame of which contains the angle increment value of each joint.

[0123] In some embodiments, the end effector pose increment sequence is determined based on the local motion coordinate system associated with the current observation information.

[0124] A local motion coordinate system is a three-dimensional reference coordinate frame that is fixed to the local geometry of the manipulated object in the current observation information. The origin and axes of this coordinate frame are independent of the robot's world coordinate system and base coordinate system, but are dynamically updated as the pose of the manipulated object changes. The pose increment of the end effector is calculated and expressed in this local motion coordinate system.

[0125] In some embodiments, the processor can dynamically update the local motion coordinate system in each control cycle of the short-time action block execution. For example, before the start of each control cycle, the current pose of the manipulated object is re-detected and the origin and axis of the local motion coordinate system are updated. The actual pose of the end effector in the previous frame is transformed into the updated local motion coordinate system, and then the pose increment of the next frame is calculated based on the updated local motion coordinate system.

[0126] Further explanation regarding the determination of the end effector pose increment sequence can be found below. Figure 5 And its related descriptions.

[0127] In some embodiments, the joint angle increment sequence is obtained by inverse kinematics solving based on the end effector pose increment sequence.

[0128] Inverse kinematics (IK) is the process of calculating the joint angles of a robotic arm in reverse order based on the desired pose of the end effector. This process involves accumulating the pose increments of each frame in the end effector pose increment sequence to the current actual end effector pose to obtain the desired absolute end effector pose for each frame. Then, the corresponding joint angle values ​​are calculated for each frame of desired absolute end effector pose. Finally, the joint angle values ​​of adjacent frames are subtracted to obtain the joint angle increment sequence.

[0129] In some embodiments, the origin and axis of the local motion coordinate system are determined by key local structures of the object being manipulated by the current task node. Key local structures include at least one of a folding axis, a gripping point, a contact point, or a handle position.

[0130] In some embodiments, the processor determines the local motion coordinate system by: performing instance segmentation and keypoint detection on the image and depth data in the current observation information to extract the category of the key local structure and its position coordinates and orientation in three-dimensional space from the detection results; when the key local structure is a grasping point, the local motion coordinate system is constructed by using the three-dimensional position of the grasping point as the origin, the normal direction of the surface of the object at the grasping point as the Z-axis, the principal direction of the tangent plane of the surface of the object at the grasping point as the X-axis, and determining the Y-axis using the right-hand rule. When the key local structure is a folding axis, the local motion coordinate system is constructed by using the geometric center of the folding axis as the origin, the axial direction of the folding axis as the Z-axis, and the direction perpendicular to the folding axis and located in the plane of the object as the X-axis.

[0131] In this embodiment, by switching the motion reference frame of the end effector from the fixed robot base coordinate system to a dynamic local motion coordinate system that is fixed to the key local structure of the manipulated object, the motion parameters of the robot are no longer affected by the translation or rotation of the manipulated object in world space when performing local interactive tasks such as folding the folding axis, grasping the gripping point, pressing the contact point, or holding the handle. This significantly improves the generalization ability and execution robustness of tasks such as folding clothes, opening doors, operating drawers, and grasping cup handles under different placement postures.

[0132] Step 404: Combine the continuous multi-frame motion parameters in time sequence to obtain a short-time action block for the embodied robot to execute the current task node.

[0133] In some embodiments, the processor appends the timing index number of the frame in the action block to each frame of the action parameters of multiple consecutive frames; sorts all frames according to the timing index number in ascending order, and encapsulates the action parameters of each frame into a structured action block data packet, which also includes the action block start timestamp and the action block duration. After encapsulation, a short-time action block is obtained.

[0134] In some embodiments, the processor may also perform timing consistency smoothing on the motion parameters of consecutive frames before combining short-time action blocks, including detecting whether there is a sudden change in the pose increment of the end effector between adjacent frames, and if there is a sudden change, performing linear interpolation smoothing on the pose increment before and after the frame with the sudden change, and then combining the smoothed motion parameters of consecutive frames into short-time action blocks in timing.

[0135] In some embodiments, after obtaining the short-time action block and before executing the short-time action block, the processor may also perform control adaptation processing on the short-time action block to obtain executable control instructions; the executable control instructions are used to control the android to execute the action corresponding to the short-time action block.

[0136] Control adaptation processing includes at least one of coordinate system transformation processing, inverse kinematics solution processing, trajectory planning processing, velocity limitation processing, acceleration limitation processing, joint limit check processing, and collision detection processing.

[0137] Executable control instructions refer to the set of low-level control signals generated by adapting the abstract motion parameters in short-time action blocks to low-level physical constraints and format conversion, which can be directly sent to the joint drivers and actuators of the embodied robot.

[0138] In some embodiments, the processor obtains executable control instructions including: performing coordinate system transformation processing on the end effector pose increment in the short-time action block, transforming the pose increment from the local motion coordinate system to the robot base coordinate system to obtain the global target pose; performing inverse kinematics solving processing on the global target pose, and using a numerical iterative algorithm to calculate the target angles of each joint of the robotic arm; performing joint limit check processing and collision detection processing on each joint target angle obtained by the solution in sequence, and marking the action parameters of the frame as unexecutable if the joint limit check or collision detection fails; performing trajectory planning processing on the joint angle sequence that passes the verification to generate a smooth trajectory in the joint space, and then performing velocity limit processing and acceleration limit processing on the smooth trajectory, and using the processed joint angle-velocity-acceleration sequence as executable control instructions.

[0139] In this embodiment, the gap between the current state and the target state is explicitly quantified through state difference features. The gap information is then fused with the end effector pose and task semantics and input into the low-level action policy network. This allows the action parameters generated by the network to directly approximate the target state and have a clear basis for closed-loop operation. The method of outputting multiple consecutive frames of action parameters into short-time action blocks at one time retains a flexible decision window for real-time feedback adjustment and deviation detection in the subsequent execution process, effectively improving the response efficiency of the embodied execution layer and the task fit of the action generation.

[0140] Figure 5 This is an exemplary flowchart illustrating the determination of an end effector pose increment sequence according to some embodiments of this application. In some embodiments, Figure 5 The process shown can be executed by an electronic device, such as a processor.

[0141] Step 501: Perform real-time semantic key point detection on the target operation object in the current observation information to obtain the position information and attitude orientation information of the key local structure of the target operation object in the current camera coordinate system.

[0142] The current camera coordinate system refers to the three-dimensional reference frame of the imaging space defined by the vision sensor mounted on the android. The origin of this coordinate frame is located at the optical center of the camera, and the three coordinate axes extend along the horizontal and vertical directions of the camera's imaging plane and the optical axis, respectively.

[0143] The location information of key local structures refers to the three-dimensional spatial coordinates of the geometric center or feature points of key local structures in the current camera coordinate system.

[0144] The attitude orientation information of a key local structure refers to the three-dimensional orientation vector or rotation matrix of the orientation axis of the key local structure in the current camera coordinate system.

[0145] Real-time semantic keypoint detection refers to the semantic-level keypoint localization of target operation objects in the currently observed information within each control cycle. The category of keypoint localization corresponds to the operation semantics of the current task node. For example, the folding axis corresponds to the endpoint of the crease line of clothing, the gripping point corresponds to the center of the cup handle, the contact point corresponds to the center of the pressing position, or the handle position corresponds to the end of the door handle.

[0146] In some embodiments, the processor obtains position information and attitude orientation information by: inputting the image from the current observation information into a deep learning detection network fine-tuned by a keypoint detection task, the detection network simultaneously outputting the key local structure category, the two-dimensional pixel coordinates of the key local structure on the image plane, and the two-dimensional orientation angle of the key local structure; back-projecting the two-dimensional pixel coordinates to the current camera coordinate system based on the two-dimensional pixel coordinates and the depth value at the corresponding position in the current observation information to obtain three-dimensional position coordinates; and calculating a three-dimensional attitude orientation vector based on the two-dimensional orientation angle and the local plane normal vector in the current camera coordinate system, wherein the three-dimensional position coordinates and the three-dimensional attitude orientation vector constitute the position information and attitude orientation information.

[0147] Step 502: Using the position information of the key local structure as the origin of the local motion coordinate system and the orientation information of the key local structure as the axis of the local motion coordinate system, construct the local motion coordinate system.

[0148] In some embodiments, the processor constructs a local motion coordinate system by: using the three-dimensional coordinate values ​​contained in the position information of the key local structure as the origin coordinates of the local motion coordinate system; using the three-dimensional orientation vector contained in the attitude orientation information of the key local structure as the Z-axis direction of the local motion coordinate system; determining a reference direction orthogonal to the Z-axis as the X-axis direction using the gravity and support plane normal vector in the current observation information as well as the X-axis direction; and using the cross product of the Z-axis and the X-axis as the Y-axis direction, with the origin, X-axis, Y-axis, and Z-axis together forming a complete local motion coordinate system.

[0149] Step 503: Transform the global end effector pose of the embodied robot from the world coordinate system to the local motion coordinate system to obtain the local reference pose.

[0150] Global end effector pose refers to the set of three-dimensional position and three-dimensional orientation of the end effector in the world coordinate system.

[0151] The world coordinate system is a predefined fixed global reference frame, usually with a fixed calibration point on the robot base or in the environment as the origin.

[0152] Local reference pose refers to the end effector pose expressed in the local motion coordinate system after mapping the global end effector pose from the world coordinate system to the local motion coordinate system through a coordinate transformation matrix.

[0153] In some embodiments, the processor obtains the local reference pose by: constructing a homogeneous transformation matrix from the world coordinate system to the local motion coordinate system based on the coordinates of the origin of the local motion coordinate system in the world coordinate system and the direction vectors of each axis of the local motion coordinate system in the world coordinate system; performing a left multiplication operation on the homogeneous pose matrix corresponding to the global end effector pose to obtain the homogeneous pose matrix of the end effector in the local motion coordinate system; and extracting the three-dimensional translation and three-dimensional rotation components from the homogeneous pose matrix as the local reference pose.

[0154] Step 504: Calculate the desired local target pose of the end effector in the local motion coordinate system based on the intermediate target representation and the current observation information.

[0155] The desired local target pose refers to the target position and orientation that the end effector should reach after completing the current task node, expressed in the local motion coordinate system. The desired local target pose can be obtained by jointly calculating the target state described by the intermediate target representation and the current state described by the current observation information in the local motion coordinate system.

[0156] In some embodiments, the processor calculates the desired local target pose by: extracting the target position and target orientation of the desired key local structure of the target operation object after the current task node is executed from the intermediate target representation; using the target position as the target position component in the desired local target pose and the target orientation as the target attitude component in the desired local target pose; and combining the target position component and the target attitude component into the desired local target pose.

[0157] Step 505: Based on the difference between the desired local target pose and the local reference pose, generate the end effector pose increment sequence in the local coordinate system.

[0158] An end effector pose increment sequence refers to an ordered set of pose increments from multiple consecutive frames in a local motion coordinate system, representing the process by which the end effector gradually approaches the desired local target pose from the current local reference pose.

[0159] In some embodiments, the processor generates an end effector pose increment sequence by: calculating a position difference vector and an attitude difference rotation matrix between the desired local target pose and the local reference pose; uniformly dividing the position difference vector according to the number of frames to obtain the position increment of each frame according to a preset action block frame number and a maximum step size limit per frame; uniformly interpolating the attitude difference rotation matrix according to the number of frames to obtain the attitude increment of each frame; and combining the position increment and attitude increment of each frame in a temporal sequence to obtain an end effector pose increment sequence in the local coordinate system.

[0160] In this embodiment, the position and orientation of the key local structure of the target object in the current camera coordinate system are dynamically tracked by real-time semantic key point detection. The entire pose calculation and incremental generation process of the end effector is uniformly migrated to the local motion coordinate system fixed to the key local structure. This makes the pose increment sequence naturally aligned with the local geometric features of the target object and is not affected by the overall displacement or rotation of the target object in world space. This effectively ensures the spatial consistency of the robot's motion generation and its cross-pose generalization ability in local interaction tasks.

[0161] Figure 6 This is an exemplary flowchart illustrating the updating of task execution data according to some embodiments of this application. In some embodiments, Figure 6 The process shown can be executed by an electronic device, such as a processor.

[0162] Step 601: Obtain the actual observation information after executing the short-time action block.

[0163] Actual observation information refers to the real-time environmental perception data re-collected by the embodied robot through its body sensors after it has completed all the actions corresponding to the short-term action block.

[0164] In some embodiments, the processor acquires actual observation information by: sending the action parameters of the last frame of the short-time action block to the servo driver and waiting for it to finish executing, synchronously acquiring the RGB image and depth map of the current frame through a vision sensor, acquiring contact force feedback data at the end effector through a force sensor, and using the RGB image, depth map and contact force feedback data together as actual observation information.

[0165] Step 602: Determine the actual observation representation based on the actual observation information, and obtain the similarity score between the actual observation representation and the intermediate target representation.

[0166] Actual observation representation refers to a high-dimensional numerical representation of the actual state of the scene after the execution of a short-duration action block, obtained by extracting features from actual observation information. The actual observation representation and the intermediate target representation reside in the same feature space to enable direct comparison.

[0167] Similarity score is a continuous numerical value used to quantify the degree of matching or consistency between the actual observed representation and the intermediate target representation. The higher the score, the closer the actual execution result is to the expected target state.

[0168] In some embodiments, the processor obtains the similarity score by: inputting the image from the actual observation information into a visual encoder, extracting the actual visual feature vector corresponding to the actual observation information as the actual observation representation; calculating the cosine similarity between the actual observation representation and the intermediate target representation, and determining the cosine similarity value as the similarity score.

[0169] Step 603: If the similarity score is greater than or equal to the preset completion threshold, then the current task node is determined to have been executed successfully, and the node execution status of the embodied robot executing the current task node is updated to the execution success status.

[0170] The preset completion threshold refers to a pre-set similarity score threshold used to determine whether the current task node has been successfully completed. The preset completion threshold can be set according to the accuracy requirements of the task type, the measurement noise level of the sensor, and the complexity of the scene.

[0171] The execution success status refers to the node execution status value in the task execution data used to mark that the current task node has been successfully completed and meets the completion conditions.

[0172] In some embodiments, the processor determines that the execution was successful and updates the status by: comparing the similarity score with a preset completion threshold, which is set differently depending on the operation object, for example, 0.95 for high-precision insertion and removal tasks and 0.85 for coarse placement tasks; when the similarity score is greater than or equal to the preset completion threshold, the processor determines that the current task node has been executed successfully and updates the node execution status field corresponding to the task node in the task execution data to the execution success status.

[0173] Step 604: If the similarity score is less than the preset completion threshold, then update the node execution status of the current task node to the execution failure status.

[0174] The execution failure status refers to the node execution status value in the task execution data used to mark the current task node's failure to meet the target status specified by the completion condition information, and which requires triggering the subsequent failure recovery mechanism.

[0175] In some embodiments, the processor determines that the execution has failed and updates the status by: when the similarity score is less than a preset completion threshold, the processor determines that the current task node has failed to execute and updates the node execution status field corresponding to the task node in the task execution data to the execution failure status.

[0176] In this embodiment, by re-collecting actual observation information and calculating the similarity score between it and the intermediate target representation after the short-term action block is executed, the abstract target state representation is transformed into a quantifiable success or failure criterion. This enables the processor to obtain a clear success or failure signal at the end of each task node, avoiding the deficiency of lacking stage-based criteria after the action is executed in the end-to-end method. This provides a reliable decision entry point for the node-by-node advancement of the structured task graph or the hierarchical recovery after failure.

[0177] Figure 7This is an exemplary flowchart illustrating hierarchical replanning based on deviation values, according to other embodiments of this application. In some embodiments, Figure 7 The process shown can be executed by an electronic device, such as a processor.

[0178] In some embodiments, after updating the node execution status of the current task node to an execution failure status, the processor may perform the following operations: Step 701: Calculate the deviation between the actual observation representation and the intermediate target representation.

[0179] The deviation measure is a single numerical metric used to quantify the degree of difference between the actual observed representation and the intermediate target representation in the feature space. The deviation measure reflects the degree of deviation between the actual state achieved after executing a short action block and the expected target state.

[0180] In some embodiments, the processor calculates the deviation value by: treating the actual observation representation and the intermediate target representation as two vectors in the feature space, calculating the Euclidean distance between the two vectors, and determining the value of the Euclidean distance as the deviation value. The larger the deviation value, the more serious the deviation of the actual execution result from the expected target state.

[0181] In some embodiments, the processor may also use the weighted sum of the channel-by-channel absolute differences between the actual observation representation and the intermediate target representation as the deviation value. The weight coefficient of each channel is predetermined according to the sensitivity of the feature dimension corresponding to the channel to the success or failure of the task. Feature channels with high sensitivity are given larger weight coefficients, and feature channels with low sensitivity are given smaller weight coefficients.

[0182] Step 702: If the deviation value is less than the first preset threshold, then the action replanning mode is triggered, and the action replanning mode returns to the step of determining the short-term action block for the embodied robot to execute the current task node based on the current observation information, the robot state data and the intermediate target representation.

[0183] The first preset threshold is a pre-set critical value for the deviation between the action layer deviation and the target layer deviation. The value of the first preset threshold can be set according to the task execution accuracy requirements, sensor noise level, and control cycle duration. When the deviation value is less than the first preset threshold, it indicates that the target state of the current task node is still reasonable, and the execution failure is due to the generated action block being disturbed or a tracking error occurring during execution.

[0184] Action replanning mode refers to the retry processing mode entered when the current task node is determined to have failed but the deviation value is small. In action replanning mode, the structured task graph and intermediate target representation remain unchanged, and only short-term action blocks are regenerated to approximate the same intermediate target representation.

[0185] In some embodiments, when the deviation value is less than a first preset threshold, the processor increments the historical retry count of the current task node by one and checks whether the historical retry count is less than a preset first retry limit. If the historical retry count is less than the first retry limit, the action replanning mode is triggered. In the action replanning mode, the structured task graph and intermediate target representation remain unchanged, and the end effector is moved to the initial safe pose of the current task node according to the preset safe backtracking trajectory. Then, the process returns to the step of determining the short-term action block in step 104 to regenerate a new short-term action block. If the historical retry count has reached the first retry limit, the processor automatically upgrades the processing mode to the target replanning mode.

[0186] Step 703: If the deviation value is greater than or equal to the first preset threshold and less than the second preset threshold, then the target replanning mode is triggered, and the process returns to the step of determining the intermediate target representation based on the current task node, the current observation information and the robot state data through the preset world model layer in the task execution model.

[0187] The second preset threshold is a pre-set critical value for the deviation amount used to distinguish between target layer deviation and task layer deviation. The second preset threshold is greater than the first preset threshold. When the deviation amount is between the first and second preset thresholds, it indicates that the target state of the current task node is no longer suitable for the current scenario or the target state is unreachable. The reason for the execution failure is that there is a structural mismatch between the intermediate target representation generated by the world model layer and the actual physical scenario.

[0188] The goal replanning mode refers to the goal correction process initiated when the current task node is determined to have failed and the deviation is at a moderate level. In this mode, the structured task graph remains unchanged, but the corrected intermediate goal representation needs to be regenerated using the experience gained from the previous failure.

[0189] In some embodiments, when the deviation value is greater than or equal to a first preset threshold and less than a second preset threshold, the processor inputs the current actual observation representation and the short-term action block trajectory data executed this time as a negative feedback signal to the condition constraint encoding module of the preset world model layer. The negative feedback signal is used to constrain the world model layer to reduce the weight of candidate states with high similarity to the current actual observation representation when regenerating the predicted state distribution. Then, it returns to the step of determining the intermediate target representation in step 103, and the preset world model layer regenerates the corrected intermediate target representation based on the original input information and the newly added negative feedback signal.

[0190] Step 704: If the deviation value is greater than or equal to the second preset threshold, the task replanning mode is triggered, and the process returns to the step of determining the structured task graph based on the user instruction and the current observation information.

[0191] Task replanning mode refers to the task-level replanning process that is initiated when the current task node is determined to have failed and the deviation reaches a severe level. In task replanning mode, the structured task graph is deemed no longer suitable for the current environmental or robot state, and a complete execution plan needs to be reconstructed starting from the task planning layer.

[0192] In some embodiments, when the deviation value is greater than or equal to a second preset threshold, the processor inputs the execution failure status of the current task node, the actual observation representation, and the deviation value as failure evidence into the context memory of the task planning layer; clears the execution plans of the current task node and all its unexecuted successor task nodes in the current structured task graph, and retains the execution result data of historically executed task nodes; then returns to the step of determining the structured task graph in step 102, whereby the task planning layer regenerates an updated structured task graph containing alternative execution paths based on the original user instructions, the current actual observation information, historical execution records, and failure evidence.

[0193] In this embodiment, by triggering action replanning mode, target replanning mode, and task replanning mode respectively according to different levels of deviation after the task node execution fails, the robot can autonomously select an appropriate recovery strategy according to the severity of the failure. When the deviation is small, only the action execution trajectory needs to be adjusted to quickly retry while retaining the original target and task graph. When the deviation is moderate, the deviation at the target layer is corrected by regenerating a corrected intermediate target representation through failure feedback. When the deviation is severe, an alternative path is replanned at the task graph level to cope with environmental changes. A clear state progression relationship and a retry upper limit protection mechanism are formed between the three modes, which effectively avoids infinite loops caused by blind retries and significantly improves the failure recovery capability and execution robustness of long-term multi-stage tasks in complex real-world environments.

[0194] Figure 8 This is an exemplary flowchart illustrating the updating of a structured task graph according to other embodiments of this application. In some embodiments, Figure 8 The process shown can be executed by an electronic device, such as a processor.

[0195] Step 801: Calculate the embedding space feature distance between the actual observation representation and the intermediate target representation, and determine the embedding space feature distance as the deviation value.

[0196] Embedding space feature distance refers to the distance value obtained by calculating the distance between the actual observation representation and the intermediate target representation in a common high-dimensional feature embedding space using a metric function.

[0197] Embedding space refers to the same high-dimensional numerical feature space to which the actual observation representation and the intermediate target representation are jointly mapped. In this space, semantically similar feature vectors are clustered together, while semantically dissimilar feature vectors are far apart.

[0198] In some embodiments, the processor determines the embedding space feature distance as a deviation value by: projecting the actual observation representation and the intermediate target representation into the embedding space through a linear projection layer to obtain the actual embedding vector and the target embedding vector; calculating the Euclidean distance between the actual embedding vector and the target embedding vector, and determining the value of the Euclidean distance as the deviation value.

[0199] Step 802: Obtain the historical retry count of the current task node.

[0200] The history retry count is an integer value used to record the number of failed retries that the current task node has undergone since its first execution. The history retry count is initialized to zero when the task node first starts execution and is incremented by one each time the action replanning mode is triggered and the process returns to the step of determining a short-term action block.

[0201] In some embodiments, the processor reads the retry count field associated with the node identifier of the current task node from the dynamic recording area of ​​the task execution data, and reads the integer value stored in the retry count field as the historical retry count; if the retry count field has not been initialized, it returns the value zero as the historical retry count.

[0202] Step 803: If the deviation value is less than the first preset threshold and the historical retry count is less than the first retry limit, then the action replanning mode is triggered.

[0203] In motion replanning mode, the structured task graph and the intermediate target representation remain unchanged. The end effector of the android is reset to the initial safe pose corresponding to the current task node, and the process returns to the step of determining the short-term action block for the android to execute the current task node. At the same time, the historical retry count is incremented by one.

[0204] The first retry limit refers to the maximum number of retries allowed for the current task node in action replanning mode, which is set in advance.

[0205] The initial safe pose refers to the preset pose in which the end effector is located before the current task node begins execution, ensuring a safe, collision-free position and facilitating the re-initiation of the operation.

[0206] In some embodiments, when the deviation value is less than a first preset threshold and the historical retry count is less than a first retry upper limit, an action replanning mode is triggered. In this mode, the processor keeps the current structured task graph and intermediate target representation completely unchanged, reads the preset initial safe pose of the current task node from the task execution data and generates a fast backtrack trajectory from the current position of the end effector to the initial safe pose, controls the end effector to move along the fast backtrack trajectory to the initial safe pose, and then returns to the step of determining the short-term action block in step 104, while incrementing the historical retry count of the current task node by one.

[0207] Step 804: If the deviation value is greater than or equal to the first preset threshold and less than the second preset threshold, or if the historical retry count reaches the first retry limit, then the target replanning mode is triggered.

[0208] In the target replanning mode, the structured task graph remains unchanged. The current observation information, the failed execution trajectory data of the current task node, and the historical retry count are input as negative feedback signals to the preset world model layer to constrain the preset world model layer to regenerate the corrected intermediate target representation. Based on the corrected intermediate target representation, the process returns to the step of determining the short-term action block for the embodied robot to execute the current task node, while resetting the historical retry count.

[0209] Failed execution trajectory data refers to the complete time series record of the entire pose sequence traversed by the end effector of the embodied robot from the start of execution at the current task node until it is judged as an execution failure, as well as the joint states and gripper opening and closing degrees in each control cycle.

[0210] Negative feedback signals refer to encoding failed execution trajectory data and historical retry counts as punitive constraints and inputting them into the feature input of a preset world model layer. This is used to reduce the tendency of the world model layer to generate candidate target states similar to those of failed trajectories.

[0211] The revised intermediate target representation refers to the new target state expression regenerated by the preset world model layer after receiving negative feedback signals.

[0212] In some embodiments, when the deviation value is between a first preset threshold and a second preset threshold, or when the historical retry count reaches a first retry limit, a target replanning mode is triggered. In this mode, the processor keeps the current structured task graph unchanged and encodes the current observation information, the failed execution trajectory data recording all end effector poses from the start of execution to the failure of the current task node, and the current value of the historical retry count into a negative feedback constraint vector. The negative feedback constraint vector is then concatenated with the task embedding vector and input into the preset world model layer, so that the world model layer actively suppresses candidate states similar to the end regions reached by the failed trajectories when generating the predicted state distribution. After the world model layer regenerates the corrected intermediate target representation, the historical retry count of the current task node is reset to zero, and the process returns to the step of determining the short-term action block in step 104.

[0213] Step 805: If the deviation value is greater than or equal to the second preset threshold, or if the number of consecutive corrections of the preset world model layer in the target replanning mode reaches the second retry limit, then the task replanning mode is triggered.

[0214] In the task replanning mode, the execution failure status of the current task node, the actual observation characteristics, and the deviation value are input as failure evidence to the task planning layer. This drives the task planning layer to perform a global consistency check on the preconditions and dependencies of the unexecuted subsequent task nodes in the structured task graph, and regenerate an updated structured task graph containing alternative execution paths based on the check results.

[0215] The number of consecutive corrections refers to the cumulative number of times the preset world model layer regenerates the corrected intermediate target representation at the same current task node due to the target replanning mode being triggered.

[0216] The second retry limit refers to the maximum number of times the world model layer is allowed to regenerate the corrected intermediate target representation under the target replanning mode, which is a pre-defined limit.

[0217] Failure evidence refers to a triplet data packet that combines the execution failure status of the current task node, actual observation characteristics, and deviation values. Failure evidence serves as prior knowledge input for the task planning layer when re-making task decisions.

[0218] Global consistency verification refers to the integrity check process by which the task planning layer checks the preconditions of each unexecuted subsequent task node in the structured task graph to see if they still hold true in the current scenario and whether the dependencies between nodes are broken due to the failure of the current node.

[0219] An alternative execution path refers to another sequence of task nodes that starts from the current scenario state, bypasses the failed task nodes or their reasons for failure, and can ultimately achieve the task objective corresponding to the user's instruction.

[0220] The updated structured task graph refers to the new structured task graph obtained after being replanned by the task planning layer.

[0221] In some embodiments, when the deviation value is greater than or equal to a second preset threshold or the number of consecutive corrections reaches a second retry limit, a task replanning mode is triggered. In this mode, the processor combines the execution failure state of the current task node, the actual observation representation, and the deviation value into a failure evidence triple and inputs it into the memory cache of the task planning layer. The task planning layer is driven to perform a global consistency check on all subsequent task nodes that have not yet been executed after the current task node in the structured task graph. This check includes checking whether the precondition information of each subsequent task node is still true in the scenario described by the current actual observation representation and whether the dependencies between each subsequent task node have logical conflicts due to the skipping of the current node. Based on the check results, the task planning layer replans an alternative execution path that can bypass the failure cause and ultimately achieve the task goal from the current scenario state, while retaining the historically successfully executed task nodes. The new task node sequence corresponding to the alternative execution path and its node execution information are encapsulated into an updated structured task graph.

[0222] In this embodiment, an infinite retries of a single node are prevented by introducing a comparison mechanism between the historical retry count and the first retry limit in the action replanning mode. In the target replanning mode, the failed execution trajectory data and the historical retry count are used as negative feedback signals to input the world model layer to constrain it to generate a more achievable corrected intermediate target representation. In the task replanning mode, task-level structural reconstruction is achieved through global consistency verification and alternative execution path generation. A complete recovery system from local adjustment to global reconstruction is formed among the three modes, which effectively solves the technical defects of existing end-to-end methods that cannot diagnose the problem level after execution failure and lack hierarchical recovery strategies, and significantly improves the robot's autonomous fault repair capability in long-term multi-stage tasks.

[0223] Figure 9 This is an exemplary flowchart illustrating the updating of short-time action blocks according to some embodiments of this application. In some embodiments, Figure 9 The process shown can be executed by an electronic device, such as a processor.

[0224] Step 901: Generate the current search key based on the current observation information.

[0225] The current search key refers to the feature vector representation extracted from the currently observed information and used for similarity retrieval in the memory.

[0226] In some embodiments, the processor generates the current search key by: performing block embedding processing on the image of each viewpoint using the visual encoder of the visual language action model to obtain a set of block features carrying spatial location encoding; rearranging the block features of each viewpoint into a two-dimensional grid according to their spatial location; performing average pooling downsampling on the two-dimensional grid of each viewpoint at a certain step size to reduce spatial resolution and enhance the translation invariance of features; flattening the pooled features of each viewpoint into a one-dimensional vector; concatenating the one-dimensional vectors of all viewpoints according to a preset viewpoint order; and performing normalization on the concatenated multi-view fusion vector to obtain the current search key.

[0227] Step 902: Obtain the preset success memory bank and the preset failure memory bank.

[0228] The success memory includes multiple prior candidate successful actions that were executed successfully in the past, and the failure memory includes multiple prior candidate failure actions that were executed unsuccessfully in the past.

[0229] A success memory is a dataset used to store key observational action segments that drive a robot's task progress through its historical successful trajectories. The success memory includes multiple candidate successful action priors from past successful executions. A candidate successful action prior is a single historical action segment stored in the success memory; each action segment contains a sequence of robot actions over multiple consecutive time steps, such as gripper displacement, rotation angle, and opening / closing control variables.

[0230] A failure memory is a dataset used to store observed action segments in a robot's historical trajectory that caused significant regressions in task progress. The failure memory includes multiple candidate failure action priors from past executions. Each candidate failure action prior refers to a single failed action segment stored in the failure memory and its corresponding risk score.

[0231] In some embodiments, obtaining preset success and failure memories includes loading pre-built memory files from an external storage medium.

[0232] Step 903: Based on the current search key, determine the target successful action prior associated with the current observation information from the successful memory, and determine the target confidence corresponding to the target successful action prior based on the current search key.

[0233] Prior successful actions refer to reference action templates retrieved from the success memory to guide the generation of the current action.

[0234] Target confidence score is a quantitative score used to characterize the prior reliability of actions leading to the success of a target.

[0235] In some embodiments, the processor determines the target successful action prior and target confidence by: calculating the cosine similarity between the current search key and the historical search key corresponding to each candidate successful action prior in the success memory; selecting the top N similarity values ​​(N is a positive integer, such as 5, 10, 20, etc.) as an initial selection set, and discarding weakly related candidates with similarity values ​​below a first threshold; for candidate action blocks in the initial selection set, calculating the multivariate dynamic time-normalized distance between every two candidate action blocks; for each candidate action block, using the median distance between it and other candidate action blocks as the trajectory inconsistency score of that candidate, and discarding candidates with inconsistency scores greater than a preset outlier threshold; and for the remaining candidate action blocks, performing a weighted summation using the normalized value of the similarity corresponding to each candidate scaled by a temperature coefficient as the weight, to obtain the target successful action prior.

[0236] In other embodiments, the processor's determination of the target's successful action prior and target confidence may further include: performing batch matrix multiplication operations on the current search key and historical search keys in the success memory to obtain all similarities at once; using an approximate nearest neighbor index structure to accelerate retrieval and returning the top N candidates with the highest similarity, without calculating all candidates; using Euclidean distance instead of multivariate dynamic time warping distance to determine trajectory consistency for the returned candidates, thereby reducing computational complexity; and aligning the remaining candidate action blocks according to timestamps and taking the median of each dimension to obtain the target's successful action prior.

[0237] Step 904: Based on the current search key and the failure memory, determine the failure exclusion item.

[0238] Target failure exclusion refers to guiding information used during action generation to push the generated trajectory away from historical failure patterns. Target failure exclusion can include a set of candidate failure exclusions, the exclusion weight of each candidate, and the failure relevance of the current state relative to the failure memory.

[0239] In some embodiments, the processor determines the target failure exclusion item by: calculating the cosine similarity between the current retrieval key and the failure retrieval key corresponding to each candidate failure action in the failure memory; selecting the top N candidates with the highest similarity as initial failure items in descending order of similarity; determining whether the similarity of each initial failure item is greater than a second threshold, and determining the initial failure items with similarity greater than the second threshold as target failure items, wherein the second threshold is used to filter failure memories that are weakly related to the current state to prevent false triggering; determining the historical failure action blocks corresponding to each target failure item and their corresponding failure risk scores as candidate failure exclusion items; determining the exclusion weight corresponding to each candidate based on the normalized value of the similarity corresponding to each candidate failure exclusion item after scaling by a temperature coefficient; determining the failure relevance of the current state relative to the failure memory based on the difference between the average similarity corresponding to each candidate failure exclusion item and the second threshold after scaling to the range of 0 to 1 by a risk normalization coefficient; and jointly determining the candidate failure exclusion items, the exclusion weights corresponding to each candidate failure exclusion item, and the failure relevance as target failure exclusion items.

[0240] Step 905: Update the short-term action block based on the prior knowledge of the target's successful action, the target's confidence level, and the failure exclusion term.

[0241] Updating short-term action blocks involves merging the original short-term action blocks generated by the low-level action policy network with the action blocks in the prior knowledge of the target's successful actions, and then applying a failure exclusion constraint to the updated short-term action blocks so that they inherit the beneficial directions of historical successful experiences while avoiding the trajectory regions of historical failures.

[0242] In some embodiments, the processor updates the short-time action block by: using the target confidence as a fusion weight, performing a frame-by-frame weighted average of the action parameters of each frame in the original short-time action block and the corresponding frame action parameters in the prior knowledge of the target successful action according to the fusion weight, to obtain a preliminary fused action block; adding a failure rejection term as an additional penalty term to each frame action parameter of the preliminary fused action block, so that the updated action parameters are shifted in the action space in a direction away from the historical failure trajectory region, and using the action block after adding the failure rejection term as the updated short-time action block.

[0243] In this embodiment, by retrieving historical successful experiences similar to the current scenario from the success memory bank as positive guidance before the action is executed, and calculating failure exclusion terms from the failure memory bank as negative constraints, the short-term action block that is finally executed can inherit the effective patterns of historical successful trajectories and avoid known failure action areas. The introduction of target confidence allows the processor to adaptively adjust the fusion strength according to the matching degree between historical experience and the current scenario, which effectively improves the action execution quality and success rate of the embodied robot in repetitive task scenarios.

[0244] The task execution model shown in some embodiments of this application can be a composite structure of a visual-language-action model and a world model. The visual-language-action (VLA) model includes a task planning layer and an embodied execution layer.

[0245] The inputs to the task execution model include user instructions, environmental observations, historical memories, constraints, and robot capability boundaries. Environmental observations may include RGB images, depth maps, point clouds, semantic segmentation, object detection, and haptic or force feedback; robot states may include manipulator joint angles, end effector poses, gripper states, chassis states, and reachability.

[0246] The model can be divided into three layers: The first layer is the task planning layer. This layer answers the questions of "what to do and how to decompose it," transforming user instructions and environmental context into a structured task graph. Each node in the task graph can contain a subtask name, operation object, preconditions, completion conditions, failure handling methods, and node dependencies. This layer outputs not low-level actions, but rather an executable, verifiable, and replannable task structure.

[0247] The second layer is the intermediate target layer, also known as the world model layer. This layer answers the questions of "what state to achieve and how to define success," and its core is the world model. The intermediate target layer does not simply receive the complete task graph, but rather receives the current task node, current observations, robot state, scene memory, and execution history, and generates the target world state after the current step is completed.

[0248] The intermediate target representation can be a sub-target image, or a target state map, semantic target, 3D target state, object pose, keypoints, segmentation mask, contact point, gripping point, placement point, target region mask, affordance representation, or latent space representation. This representation can also correspond to success conditions, such as whether the object is stably gripped, whether it is within the target region, whether the target spatial relationship is satisfied, and whether a collision has occurred.

[0249] The third layer is the embodied execution layer. This layer answers the questions of "how to do it and how to execute in a closed loop," generating short-term action blocks based on current observations, robot state, intermediate target representations, and task semantics. Action blocks can include end-effector pose increments, target poses, joint angles, joint velocities, gripper opening / closing commands, chassis velocities, or multi-robot cooperative action commands.

[0250] In some embodiments, the action blocks output by the embodied execution layer can typically be further adapted by control to be converted into control commands executable by a specific robot. The control adaptation process may include coordinate system transformation, inverse kinematics solving, trajectory planning, trajectory smoothing, velocity / acceleration limiting, joint limit checking, collision detection, and controller delivery.

[0251] See Figure 10 , Figure 10 This is an exemplary schematic diagram of the model structure of a task execution model according to some embodiments of this application.

[0252] like Figure 10 As shown, the task execution model includes HL - High-level planner, WM - World model, and LL - Low-level strategy, which correspond to the task planning layer, the world model layer, and the embodied execution layer, respectively.

[0253] The HL-high-level planner outputs subtasks based on observation information, task instructions, and historical subtasks; the WM-world model outputs intermediate action representations based on observation information and subtasks; the LL-low-level policy outputs action blocks based on observation information, subtasks, and intermediate action representations; and the executor then converts the action blocks into action execution.

[0254] Taking the task of folding clothes as an example, L1 determines whether the current stage is "flattening the clothes" or "first fold"; L2 generates the target state of the clothes after the corresponding stage is completed, such as the clothes being flattened, key points being aligned, or the folded edge reaching the target position; L3 generates actions such as grabbing, moving, folding, putting down, and flattening based on the difference between the current state and the target state.

[0255] Taking the task of putting a cup into a sink as an example, L1 generates task nodes such as locating the cup, grabbing the cup, moving it to the sink, putting it into the sink, and confirming completion; when the current node is "grabbing the cup", L2 generates target states such as the gripper approaching the handle, the gripper holding the cup stably, and the cup leaving the table; L3 generates action blocks such as approaching, adjusting the posture, closing the gripper, and lifting.

[0256] If the current observation is inconsistent with the intermediate target after execution, the system can trigger three processing methods based on the deviation type: regenerate a short-term action block when the action deviation is small; regenerate an intermediate target when the target state is no longer suitable for the current scene; or return to the task planning layer to replan the task graph when the task preconditions have changed.

[0257] The task planning layer can be implemented using a multimodal large model, a language model, a symbolic planner, a task graph generation network, or a rule engine. The output of the world model is not limited to sub-target images; it can also be 3D states, keypoints, poses, masks, affordances, semantic states, or latent space vectors. The embodied execution layer can employ VLA action models, diffusion action models, flow matching action models, policy networks, or combinations thereof.

[0258] Reference Figure 11 The diagram shown is a functional module schematic of the embodied robot control system 100 based on multi-segment task planning in this application.

[0259] The embodied robot control system 100 based on multi-segment task planning described in this application is installed in an electronic device. Depending on the functions implemented, the embodied robot control system 100 includes an acquisition module 110, a first determination module 120, a second determination module 130, a third determination module 140, and an execution module 150. These modules can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and are stored in the memory of the electronic device.

[0260] In this embodiment, the functions of each module / unit are as follows: The acquisition module 110 is used to acquire input information, including user instructions, current observation information, and robot state data of the embodied robot. The first determining module 120 is used to determine a structured task graph based on the user instructions and the current observation information through the task planning layer in the task execution model; the structured task graph includes multiple task nodes with dependencies required to complete the user instructions; The second determining module 130 is used to determine the current task node from the structured task graph, and based on the current task node, the current observation information and the robot state data, determine the intermediate target representation through the preset world model layer in the task execution model; the intermediate target representation is used to represent the target state that should be reached after the current task node is completed. The third determining module 140 is used to determine, through the embodied execution layer in the task execution model, a short-term action block for the embodied robot to execute the current task node based on the current observation information, the robot state data, and the intermediate target representation; The execution module 150 is used to execute the action corresponding to the short-time action block, and update the task execution data of the embodied robot according to the matching of the execution result of the short-time action block with the intermediate target representation.

[0261] The specific implementation of the embodied robot control system based on multi-segment task planning in this application is largely the same as the specific implementation of the embodied robot control method based on multi-segment task planning described above, and will not be repeated here.

[0262] Reference Figure 12 The diagram shown is a schematic representation of a preferred embodiment of the electronic device of this application.

[0263] The electronic device includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. The memory 113 is used to store computer programs, such as a holographic robot control program based on multi-segment task planning; In some embodiments, the processor 111 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 111 is typically used to control the overall operation of the electronic device, such as performing data interaction or communication-related control and processing. In this embodiment, the processor 111 is used to run program code stored in the memory 113 or process data.

[0264] The communication interface 112 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The communication interface 112 may also be used to establish a communication connection between the electronic device and other electronic devices.

[0265] The memory 113 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 113 may be an internal storage unit of the electronic device, such as the hard disk or RAM of the electronic device. In other embodiments, the memory 113 may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. of the electronic device. Of course, the memory 113 may include both internal storage units and external storage devices of the electronic device. In this embodiment, the memory 113 is typically used to store the operating system and various computer programs installed on the electronic device, such as the program code of a holographic robot control program based on multi-segment task planning. In addition, the memory 113 can also be used to temporarily store various types of data that have been output or will be output.

[0266] Figure 12 Only an electronic device having a processor 111, a communication interface 112, a memory 113 and a communication bus 114 is shown. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0267] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the embodied robot control method based on multi-segment task planning provided in any of the foregoing method embodiments, including: Acquire input information; the input information includes user commands, current observation information, and robot state data of the embodied robot; The task planning layer in the task execution model determines a structured task graph based on the user instructions and the current observation information; the structured task graph includes multiple task nodes with dependencies required to complete the user instructions; The current task node is determined from the structured task graph, and based on the current task node, the current observation information, and the robot state data, an intermediate target representation is determined through a preset world model layer in the task execution model; the intermediate target representation is used to represent the target state that should be reached after the current task node is completed. Based on the current observation information, the robot state data, and the intermediate target representation, the embodied execution layer in the task execution model determines a short-term action block for the embodied robot to execute the current task node. The action corresponding to the short-time action block is executed, and the task execution data of the embodied robot is updated according to the matching between the execution result of the short-time action block and the intermediate target representation.

[0268] For a detailed explanation of the above steps, please refer to the above. Figure 1 A flowchart illustrating an embodiment of an embodied robot control method based on multi-segment task planning.

[0269] Furthermore, this application also proposes a computer-readable storage medium that is both non-volatile and volatile. This computer-readable storage medium is any one or any combination of several of the following: hard disk, multimedia card, SD card, flash memory card, SMC, read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, etc. The computer-readable storage medium includes a data storage area and a program storage area. The program storage area stores a android control program based on multi-segment task planning. When executed by a processor, the android control program based on multi-segment task planning performs the following operations: Acquire input information; the input information includes user commands, current observation information, and robot state data of the embodied robot; The task planning layer in the task execution model determines a structured task graph based on the user instructions and the current observation information; the structured task graph includes multiple task nodes with dependencies required to complete the user instructions; The current task node is determined from the structured task graph, and based on the current task node, the current observation information, and the robot state data, an intermediate target representation is determined through a preset world model layer in the task execution model; the intermediate target representation is used to represent the target state that should be reached after the current task node is completed. Based on the current observation information, the robot state data, and the intermediate target representation, the embodied execution layer in the task execution model determines a short-term action block for the embodied robot to execute the current task node. The action corresponding to the short-time action block is executed, and the task execution data of the embodied robot is updated according to the matching between the execution result of the short-time action block and the intermediate target representation.

[0270] The specific implementation of the computer-readable storage medium in this application is largely the same as the specific implementation of the embodied robot control method based on multi-segment task planning described above, and will not be repeated here.

[0271] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, system, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, system, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, system, article, or method that includes that element.

[0272] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware simulation platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0273] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A control method for an embodied robot based on multi-segment task planning, characterized in that, The method includes: Acquire input information; the input information includes user commands, current observation information, and robot state data of the embodied robot; The task planning layer in the task execution model determines a structured task graph based on the user instructions and the current observation information; the structured task graph includes multiple task nodes with dependencies required to complete the user instructions; The current task node is determined from the structured task graph, and visual features are extracted from the current observation information to obtain a first feature vector; pose features are extracted from the robot state data to obtain a second feature vector; the task semantics of the current task node and the node attribute information corresponding to the current task node are encoded to obtain a task embedding vector; the node attribute information includes precondition information associated with the embodied robot; the first feature vector, the second feature vector, and the task embedding vector are input into a preset world model layer in the task execution model to obtain a predicted state distribution; the predicted state distribution is used to characterize the candidate state distribution after executing the current task node; based on the precondition information and the robot state data, the reachability of the candidate states in the predicted state distribution is evaluated to obtain the target state that should be reached after executing the current task node, and an intermediate target representation is determined based on the target state; the intermediate target representation is used to characterize the target state that should be reached after executing the current task node. Based on the current observation information, the robot state data, and the intermediate target representation, the embodied execution layer in the task execution model determines a short-term action block for the embodied robot to execute the current task node. The action corresponding to the short-time action block is executed, and the task execution data of the embodied robot is updated according to the matching between the execution result of the short-time action block and the intermediate target representation.

2. The embodied robot control method based on multi-segment task planning as described in claim 1, characterized in that, The step of determining a structured task graph through the task planning layer in the task execution model, based on the user instructions and the current observation information, includes: The task objective is obtained by performing semantic parsing processing based on the user instructions. Based on the current observation information, scene recognition processing is performed to determine the operation object information and scene constraint information; Based on the task objective, the operation object information, the scene constraint information, and the robot capability boundary information associated with the embodied robot, determine multiple sub-tasks required to complete the task objective and the logical temporal relationship between the multiple sub-tasks; Based on the multiple subtasks and the logical timing relationship, determine the multiple task nodes corresponding to the multiple subtasks; Determine the node execution information corresponding to the multiple task nodes, and construct a structured task graph based on the multiple task nodes and the node execution information.

3. The embodied robot control method based on multi-segment task planning as described in claim 2, characterized in that, Each task node in the structured task graph has corresponding node attribute information; the node attribute information includes subtask name, operation object identifier, precondition information, completion condition information, and failure handling strategy; the precondition information is used to characterize the state conditions that the embodied robot needs to meet before executing the task node, and the completion condition information is used to characterize the state conditions that the embodied robot needs to achieve after executing the task node.

4. The embodied robot control method based on multi-segment task planning as described in claim 1, characterized in that, The step of inputting the first feature vector, the second feature vector, and the task embedding vector into a preset world model layer in the task execution model to obtain the predicted state distribution includes: The spatial features of the current scene where the embodied robot is located are constructed based on the first feature vector and the second feature vector; Based on the task embedding vector, the spatial features are conditionally constrained and encoded to obtain conditional features; Based on the aforementioned conditional features, a predicted state distribution is generated through a preset world model layer in the task execution model; wherein, the predicted state distribution includes at least one of a predicted image distribution, a predicted 3D point cloud distribution, or a predicted key point thermal distribution.

5. The embodied robot control method based on multi-segment task planning as described in claim 1, characterized in that, The step of determining a short-term action block for the embodied robot to execute the current task node through the embodied execution layer in the task execution model, based on the current observation information, the robot state data, and the intermediate target representation, includes: Based on the current observation information, determine the current state representation, obtain the state difference between the current state representation and the intermediate target representation, and obtain the state difference feature; The policy input features are obtained by concatenating the state difference features, the current end effector pose in the robot state data, and the task semantics of the current task node. The policy input features are input into a preset low-level motion policy network to generate continuous multi-frame motion parameters; each frame of motion parameters includes at least one of the following: end effector pose increment, gripper opening and closing parameters, joint control parameters, and motion confidence. The continuous multi-frame motion parameters are combined in time sequence to obtain a short-time action block for the embodied robot to execute the current task node.

6. The embodied robot control method based on multi-segment task planning as described in claim 5, characterized in that, After obtaining the short-time action block, and before executing the short-time action block, the method further includes: The short-time action block is subjected to control adaptation processing to obtain executable control instructions; the executable control instructions are used to control the android to perform the action corresponding to the short-time action block. The control adaptation process includes at least one of coordinate system transformation processing, inverse kinematics solution processing, trajectory planning processing, velocity limitation processing, acceleration limitation processing, joint limit check processing, and collision detection processing.

7. The embodied robot control method based on multi-segment task planning as described in claim 5, characterized in that, The continuous multi-frame motion parameters include an end effector pose increment sequence or a joint angle increment sequence; The end effector pose increment sequence is determined based on the local motion coordinate system related to the current observation information; the joint angle increment sequence is obtained by inverse kinematics solution based on the end effector pose increment sequence; The origin and axis of the local motion coordinate system are determined by the key local structure of the object being operated on by the current task node; the key local structure includes at least one of a folding axis, a gripping point, a contact point, or a handle position.

8. The embodied robot control method based on multi-segment task planning as described in claim 7, characterized in that, The end effector pose increment sequence is determined in the following way: Real-time semantic key point detection is performed on the target operation object in the current observation information to obtain the position information and attitude orientation information of the key local structure of the target operation object in the current camera coordinate system; The local motion coordinate system is constructed by using the position information of the key local structure as the origin of the local motion coordinate system and the orientation information of the key local structure as the axis of the local motion coordinate system. The global end effector pose of the android is transformed from the world coordinate system to the local motion coordinate system to obtain the local reference pose; Based on the intermediate target representation and the current observation information, the desired local target pose of the end effector is calculated in the local motion coordinate system. Based on the difference between the desired local target pose and the local reference pose, an incremental sequence of the end effector pose in the local coordinate system is generated.

9. The embodied robot control method based on multi-segment task planning as described in claim 1, characterized in that, The task execution data includes node execution status; updating the embodied robot's task execution data based on the matching result of the short-term action block with the intermediate target representation includes: Obtain the actual observation information after executing the short-time action block; Based on the actual observation information, the actual observation representation is determined, and a similarity score is obtained between the actual observation representation and the intermediate target representation. If the similarity score is greater than or equal to the preset completion threshold, the current task node is determined to have been executed successfully, and the node execution status of the embodied robot executing the current task node is updated to the execution success status. If the similarity score is less than the preset completion threshold, the execution status of the current task node will be updated to execution failure.

10. The embodied robot control method based on multi-segment task planning as described in claim 9, characterized in that, After updating the execution status of the current task node to an execution failure status, the method further includes: Calculate the embedding space feature distance between the actual observed representation and the intermediate target representation, and determine the embedding space feature distance as the deviation value; Obtain the historical retry count of the current task node; If the deviation value is less than the first preset threshold and the historical retry count is less than the first retry limit, then the action replanning mode is triggered. In the motion replanning mode, the structured task graph and the intermediate target representation remain unchanged. The end effector of the android is reset to the initial safe pose corresponding to the current task node, and the process returns to the step of determining the short-term action block for the android to execute the current task node. At the same time, the historical retry count is incremented by one.

11. The embodied robot control method based on multi-segment task planning as described in claim 10, characterized in that, The method further includes: If the deviation value is greater than or equal to the first preset threshold and less than the second preset threshold, or if the historical retry count reaches the first retry limit, then the target replanning mode is triggered. In the target replanning mode, the structured task graph remains unchanged, and the actual observation information, the failed execution trajectory data of the current task node, and the historical retry count are input as negative feedback signals to the preset world model layer to constrain the preset world model layer to regenerate the corrected intermediate target representation. Based on the corrected intermediate target representation, the process returns to the step of determining the short-term action block for the embodied robot to execute the current task node, while resetting the historical retry count.

12. The embodied robot control method based on multi-segment task planning as described in claim 11, characterized in that, The method further includes: If the deviation value is greater than or equal to the second preset threshold, or if the number of consecutive corrections of the preset world model layer in the target replanning mode reaches the second retry limit, then the task replanning mode is triggered. In the task replanning mode, the execution failure status of the current task node, the actual observation characteristics, and the deviation value are input as failure evidence to the task planning layer. This drives the task planning layer to perform a global consistency check on the preconditions and dependencies of the unexecuted subsequent task nodes in the structured task graph, and regenerate an updated structured task graph containing alternative execution paths based on the check results.

13. The embodied robot control method based on multi-segment task planning as described in claim 1, characterized in that, Before executing the action corresponding to the short-time action block, the method further includes: Based on the current observation information, generate the current search key; Obtain a preset success memory bank and a preset failure memory bank; the success memory bank includes multiple prior candidate successful actions that were executed successfully in the past, and the failure memory bank includes multiple prior candidate failure actions that were executed unsuccessfully in the past. Based on the current retrieval key, determine the target successful action prior associated with the current observation information from the successful memory, and determine the target confidence corresponding to the target successful action prior based on the current retrieval key; Based on the current search key and the failure memory, determine the failure exclusion item; The short-term action block is updated based on the prior knowledge of the target's successful action, the target's confidence level, and the failure exclusion term.

14. A embodied robot control system based on multi-segment task planning, characterized in that, The system includes: The acquisition module is used to acquire input information, including user commands, current observation information, and robot state data of the embodied robot. The first determining module is used to determine a structured task graph based on the user instructions and the current observation information through the task planning layer in the task execution model; the structured task graph includes multiple task nodes with dependencies required to complete the user instructions; The second determining module is used to determine the current task node from the structured task graph, extract visual features from the current observation information to obtain a first feature vector; extract pose features from the robot state data to obtain a second feature vector; encode the task semantics of the current task node and the node attribute information corresponding to the current task node to obtain a task embedding vector; the node attribute information includes precondition information associated with the embodied robot; input the first feature vector, the second feature vector, and the task embedding vector into a preset world model layer in the task execution model to obtain a predicted state distribution; the predicted state distribution is used to characterize the candidate state distribution after executing the current task node; based on the precondition information and the robot state data, perform reachability evaluation on the candidate states in the predicted state distribution to obtain the target state that should be reached after executing the current task node, and determine an intermediate target representation based on the target state; the intermediate target representation is used to characterize the target state that should be reached after executing the current task node. The third determining module is used to determine, through the embodied execution layer in the task execution model, a short-term action block for the embodied robot to execute the current task node, based on the current observation information, the robot state data, and the intermediate target representation. The execution module is used to execute the actions corresponding to the short-time action block, and update the task execution data of the embodied robot according to the matching of the execution result of the short-time action block with the intermediate target representation.

Images