Humanoid robot whole-body motion planning method and related device

Abstract

This application relates to the field of robotics technology, and in particular to a method and related equipment for whole-body motion planning of humanoid robots. The method includes: determining scene semantic features and target task features through multi-source perception data, determining environmental dynamic prediction results and motion prospective features, updating full-dimensional constraints based on environmental dynamic prediction results, extracting joint coupling features at different times, extracting spatiotemporal global features and single-joint local features, generating trajectory prior information, generating an initial trajectory sequence based on the trajectory prior information, sampling noise in the initial trajectory sequence and iteratively denoising under full-dimensional constraints to obtain a full-degree-of-freedom predicted trajectory, and mapping the full-degree-of-freedom predicted trajectory to joint control commands for driving joints to perform actions. The whole-body motion planning method for humanoid robots in this application can reduce the risk of trajectory deviation from a single joint of a humanoid robot spreading to the whole body through coupling relationships, and realize long-term motion planning for humanoid robots.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method and related equipment for whole-body motion planning of humanoid robots. Background Technology

[0002] Humanoid robots have dozens of joints. The movement of local joints can change the overall center of gravity of the humanoid robot. Therefore, when executing input commands, the control system of a humanoid robot not only needs to plan the overall motion trajectory, but also needs to coordinate the actions of multiple coupled local joints.

[0003] Among related technologies, there exists a generative planning method based on deep learning, such as trajectory prediction methods based on Transformer and diffusion models. These methods generate multiple feasible trajectories through the diffusion model, then select the optimal trajectory from these trajectories using a pre-trained denoising network. The optimal trajectory is then mapped to control commands for each joint, enabling full-body control of the humanoid robot. However, in practical applications, it has been found that existing methods do not explicitly model the kinematic coupling relationships between joints. In long-term planning, small errors in a single joint can rapidly propagate to the entire body through these coupling relationships, causing the trajectory to deviate significantly from the target and making accurate long-term planning impossible.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main objective of this application is to propose a method and related equipment for whole-body motion planning of a humanoid robot, which can explicitly model the kinematic coupling relationship between joints, reduce the risk of trajectory deviation caused by the spread of small errors in a single joint to the whole body through the coupling relationship, and realize long-term motion planning of the humanoid robot.

[0006] To achieve the above objectives, one embodiment of this application provides a method for whole-body motion planning of a humanoid robot, the method comprising the following steps: In response to receiving a task instruction, the system analyzes the currently collected multi-source perception data using a multimodal model to determine the scene semantic features and target task features. By encoding scene semantic features and historical motion state data using a world model, the dynamic prediction results of the environment and the prospective features of motion are determined. The full-dimensional constraints are updated based on the environmental dynamic prediction results, and the updated full-dimensional constraints are input into the iterative denoising process. The full-dimensional constraints include obstacle avoidance constraint boundaries, human-machine safety distance constraints, ZMP stability constraint boundaries and / or dynamic torque constraint weights. The scene semantic features, motion prospective features, environmental dynamic prediction results and historical motion data are extracted by the topological graph neural network unit to obtain joint coupling features at different times. The joint coupling features and scene semantic features are extracted by the spatiotemporal cross attention unit to obtain spatiotemporal global features and single joint local features. Based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features, trajectory prior information is generated. An initial trajectory sequence is generated based on the trajectory prior information. The initial trajectory sequence is iteratively denoised according to the initial trajectory sequence, the noise time step, the spatiotemporal global features, the single joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory. The fully free-degree-of-freedom predicted trajectory is mapped to joint control commands, which are used to drive joints to perform actions.

[0007] In some embodiments, updating the full-dimensional constraints based on the environmental dynamic prediction results includes: When the environmental dynamic prediction result represents a change in the obstacle's position, the obstacle avoidance constraint boundary is updated. When the environmental dynamic prediction result represents a change in the distance between the human and the robot, the human-robot safe distance constraint is updated. When the environmental dynamic prediction results characterize changes in the support region or center of gravity, the ZMP stability constraint boundary is updated. When the environmental dynamic prediction results characterize load or speed changes, the dynamic torque constraint weights are updated.

[0008] In some embodiments, the historical motion data includes historical trajectory sequences and current joint state data. The step of extracting features from the scene semantic features, the motion prospective features, the environmental dynamic prediction results, and the historical motion data using a topological graph neural network unit to obtain joint coupling features at different times includes: Obtain the historical trajectory sequence and the current joint state data; The historical trajectory sequence, the current joint state data, the scene semantic features, the environmental dynamic prediction results, and the motion prospective features are input into the trained topological graph neural network unit; The joint coupling features at different times are obtained by aggregating information of adjacent nodes from the pre-constructed topological association graph through the topological graph neural network unit. The topological graph uses each joint degree of freedom of the humanoid robot as a node and adjacent and coupled joints in the kinematic tree as edges. The adjacency matrix corresponds one-to-one with the kinematic tree. When performing graph convolution on the topological graph, the information of adjacent nodes is aggregated only along the adjacency matrix to explicitly model the joint coupling features.

[0009] In some embodiments, the topology graph neural network unit is constructed through the following steps: Obtain the degree-of-freedom coefficients and joint connection relationships of the humanoid robot; Construct a kinematic tree structure based on the joint connection relationships; The number of nodes in the topological graph neural network unit is determined based on the degree of freedom coefficients, and the edge connection relationships between the nodes and the adjacency matrix of the topological graph neural network unit are mapped one-to-one with the kinematic tree structure.

[0010] In some embodiments, when the degree-of-freedom coefficients of the humanoid robot change, the number of nodes in the topological graph, the dimension of the adjacency matrix, the input-output dimension of the neural network unit of the topological graph, and the dimension of the fully free-degree-of-freedom predicted trajectory are adjusted according to the degree-of-freedom coefficients.

[0011] In some embodiments, the spatiotemporal cross-attention unit includes a temporal attention branch, a spatial attention branch, a cross-modal attention branch, and a single-joint feature decoupling branch; The method involves extracting features from the joint coupling features and the scene semantic features using a spatiotemporal cross-attention unit to obtain spatiotemporal global features and single-joint local features, including: The long-term temporal dependencies of the same joint at different time steps are extracted through the temporal attention branch to determine the temporal attention features; The spatial attention branch is used to extract the coupling correlation between different joints at the same moment to determine the spatial attention features; The cross-modal attention branch is used to extract the global association between the scene semantic features and the joint motion features to determine the cross-modal semantic features; The spatiotemporal global features are obtained by concatenating the temporal attention features, the spatial attention features, and the cross-modal semantic features and then mapping them through a linear layer. By decoupling the single-joint feature branch, the independent temporal motion features of each joint are extracted from the temporal attention features, and the single-joint local features are obtained after linear layer mapping.

[0012] In some embodiments, the step of iteratively denoising the initial trajectory sequence based on the initial trajectory sequence, the noise time step, the spatiotemporal global features, the single-joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory includes: The initial trajectory sequence, noise time steps, spatiotemporal global features, and single-joint local features are input into a pre-trained joint trajectory denoising network, which outputs the denoised trajectory increment. By using a pre-trained constraint embedding layer to perform hard constraint projection and soft constraint weighted correction on the denoised trajectory increment, a fully free-degree-of-freedom predicted trajectory is generated.

[0013] In some embodiments, the step of generating a fully free-degree-of-freedom predicted trajectory by performing hard-constraint projection and soft-constraint weighted correction on the denoised trajectory increment through a pre-trained constraint embedding layer includes: After each iteration, the noise trajectory output by the joint trajectory denoising network is clipped based on hard constraint projection. The gradient direction of the noise prediction is corrected based on soft constraints, and the predicted trajectory of the full degree of freedom is obtained. The updated full-dimensional constraints include dynamic torque constraints, ZMP stability constraints, obstacle avoidance constraints, and human-machine safety distance constraints.

[0014] In some embodiments, the trajectory prior information includes endpoint position constraints and full-time contact state constraints; The generation of trajectory prior information based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features includes: The spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features are input into a pre-trained contact endpoint prediction module, which outputs the endpoint position constraints and the contact state constraints. The contact endpoint prediction module is constructed based on a bidirectional gated recurrent unit (GRU) and a constrained fully connected layer.

[0015] To achieve the above objectives, another aspect of this application provides a humanoid robot whole-body motion planning device, the device comprising: The first module is used to respond to the received task instructions by analyzing the currently collected multi-source perception data through a multimodal model to determine the scene semantic features and target task features; The second module is used to encode scene semantic features and historical motion state data through a world model to determine the dynamic prediction results of the environment and the prospective features of motion. The third module is used to update the full-dimensional constraints based on the environmental dynamic prediction results, and input the updated full-dimensional constraints into the iterative denoising process. The full-dimensional constraints include obstacle avoidance constraint boundaries, human-machine safety distance constraints, ZMP stability constraint boundaries and / or dynamic torque constraint weights. The fourth module is used to extract features from the scene semantic features, the motion prospect features, the environmental dynamic prediction results and historical motion data through the topological graph neural network unit to obtain joint coupling features at different times, and to extract features from the joint coupling features and the scene semantic features through the spatiotemporal cross attention unit to obtain spatiotemporal global features and single joint local features. The fifth module is used to generate trajectory prior information based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features; generate an initial trajectory sequence based on the trajectory prior information; iteratively denoise the initial trajectory sequence according to the initial trajectory sequence, the noise time step, the spatiotemporal global features, the single joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory; and map the fully free-degree-of-freedom predicted trajectory into joint control commands, which are used to drive joints to perform actions.

[0016] To achieve the above objectives, another aspect of this application provides an electronic device, including: at least one processor; at least one memory for storing at least one program; and when the at least one program is executed by the at least one processor, causing the at least one processor to implement the method described above.

[0017] To achieve the above objectives, in another aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0018] The embodiments of this application include at least the following beneficial effects: This application provides a method and related equipment for whole-body motion planning of a humanoid robot. This method, in response to a received task instruction, analyzes the currently collected multi-source perception data using a multimodal model to determine scene semantic features and target task features. It then encodes the scene semantic features and historical motion state data using a world model to determine the environmental dynamic prediction result and motion prospective features. Based on the environmental dynamic prediction result, it updates the full-dimensional constraints and inputs the updated full-dimensional constraints into an iterative denoising process. The full-dimensional constraints include obstacle avoidance constraint boundaries, human-robot safe distance constraints, ZMP stability constraint boundaries, and / or dynamic torque constraint weights. A topological graph neural network unit extracts features from the scene semantic features, the motion prospective features, the environmental dynamic prediction result, and the historical motion data to obtain... Simultaneously, the joint coupling features are extracted using a spatiotemporal cross-attention unit, along with the scene semantic features, to obtain spatiotemporal global features and single-joint local features. Based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features, trajectory prior information is generated. An initial trajectory sequence is generated based on this trajectory prior information. Noise sampling and iterative denoising under full-dimensional constraints are performed on the initial trajectory sequence to obtain a fully free-degree-of-freedom predicted trajectory. This fully free-degree-of-freedom predicted trajectory is mapped to joint control commands. In this way, the topological graph neural network unit explicitly models the kinematic coupling relationship between joints as joint coupling features. Using these features as input when predicting the trajectory sequence reduces the risk of small errors in a single joint spreading to the entire body through the coupling relationship, thus enabling long-term motion planning for the humanoid robot. Attached Figure Description

[0019] Figure 1 This is an optional flowchart of the humanoid robot whole-body motion planning method provided in the embodiments of this application; Figure 2 yes Figure 1 A flowchart of sub-step 103 in the method shown; Figure 3 yes Figure 1 The flowchart shows the steps for determining the joint coupling features in the method shown. Figure 4 yes Figure 1 The flowchart of the construction process of the topology graph neural network unit in the method shown; Figure 5 yes Figure 1 The topological association graph in the method shown; Figure 6 yes Figure 1 The flowchart shows the steps for determining spatiotemporal global features and single-joint local features in the method shown. Figure 7yes Figure 1 The flowchart shows the steps for determining the fully free-degree-of-freedom predicted trajectory in the method shown. Figure 8 yes Figure 7 A flowchart of sub-step 702 in the method shown; Figure 9 This is a schematic diagram of the structure of the humanoid robot whole-body motion planning device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0021] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0022] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0025] With the rapid iteration of artificial intelligence and robotics technologies, humanoid robots, with their human-like body structure, strong environmental adaptability, and multi-task generalization capabilities, have become the core carriers of next-generation intelligent manufacturing and smart homes. Full-body motion planning is the core underlying technology for humanoid robots to achieve autonomous operation, directly determining the robot's task execution capability, motion stability, and scene adaptability.

[0026] Currently, existing technologies for motion planning of humanoid robots can be mainly divided into two categories: one is traditional model-based planning methods, such as Model Predictive Control (MPC) and Quadratic Programming (QP). These methods rely on accurate robot dynamics models and can only achieve local optimum planning in the short time domain. The computational load of long-time domain planning increases exponentially, which cannot meet the requirements of real-time control. The other category is generative planning methods based on deep learning, such as trajectory prediction methods based on Transformer and diffusion models. These methods have the ability to generate long sequences, but existing solutions mostly directly transfer the technical framework of autonomous driving and pedestrian trajectory prediction without deeply adapting to the specific characteristics of humanoid robots, such as strong coupling of multiple degrees of freedom, bipedal stability dependence, and strict hard constraints.

[0027] Humanoid robots have dozens of joints. The movement of local joints can change the overall center of gravity of the humanoid robot. Therefore, when executing input commands, the control system of a humanoid robot not only needs to plan the overall motion trajectory, but also needs to coordinate the actions of multiple coupled local joints.

[0028] In related technologies, generative planning methods based on deep learning, such as trajectory prediction methods based on Transformer and diffusion models, generate multiple feasible trajectories through the diffusion model. Then, a pre-trained denoising network selects the optimal trajectory from these feasible trajectories and maps the optimal trajectory to control commands for each joint, thus achieving full-body control of the humanoid robot. However, in practical applications, it has been found that existing methods do not explicitly model the kinematic coupling relationships between joints. In long-term planning, small errors in a single joint can rapidly propagate to the entire body through these coupling relationships, causing the trajectory to deviate significantly from the target and making accurate long-term planning impossible.

[0029] In view of this, this application provides a method and related equipment for whole-body motion planning of a humanoid robot. This method, in response to a received task instruction, analyzes the currently collected multi-source perception data using a multimodal model to determine scene semantic features and target task features. It then encodes the scene semantic features and historical motion state data using a world model to determine the environmental dynamic prediction result and motion prospective features. Finally, it extracts features from the scene semantic features, motion prospective features, environmental dynamic prediction result, and historical motion data using a topological graph neural network unit to obtain joint coupling features at different times. Finally, it uses a spatiotemporal cross-attention unit to analyze the joint coupling features and the scene semantic features and the target task features. Feature extraction is performed on scene semantic features to obtain spatiotemporal global features and single-joint local features. Based on the spatiotemporal global features, scene semantic features, environmental dynamic prediction results, and target task features, trajectory prior information is generated. An initial trajectory sequence is generated based on the trajectory prior information. The initial trajectory sequence is subjected to noise sampling and iterative denoising under full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory. The fully free-degree-of-freedom predicted trajectory is mapped to joint control commands. This scheme can explicitly model the kinematic coupling relationship between joints, reduce the risk of small errors in a single joint spreading to the whole body trajectory deviation through the coupling relationship, and realize long-term motion planning for humanoid robots.

[0030] The humanoid robot full-body motion planning method provided in this application relates to the field of information technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the humanoid robot full-body motion planning method, but is not limited to the above forms.

[0031] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0032] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0033] Figure 1 This is an optional flowchart of the humanoid robot whole-body motion planning method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 101 to 104.

[0034] Step 101: In response to receiving the task instruction, the currently collected multi-source perception data is analyzed through a multimodal model to determine the scene semantic features and target task features; When a user inputs a task instruction described in natural language, the multi-source perception data collected by the humanoid robot and the task instruction are input into the VLA vision-language-action multimodal large model to complete scene semantic extraction, task intent parsing and action semantic mapping, and output scene semantic features and target task features.

[0035] It is understood that the multi-source sensing data in step 101 is collected by multi-source sensors. The multi-source sensing data includes, but is not limited to, the current RGB-D visual data, whole-body joint angle data, whole-body joint angular velocity data, whole-body joint angular acceleration data, bipedal six-dimensional force sensor data, bi-arm end effector six-dimensional force sensor data, and torso IMU posture data.

[0036] The VLA Vision-Language-Action Multimodal Model comprises a data preprocessing module and a VLA multimodal semantic parsing module. The data preprocessing module performs hardware time synchronization, denoising, and standardization of multi-source perceptual data, providing compliant and standardized input data for all subsequent modules. The VLA multimodal semantic parsing module is used for parsing natural language task instructions, extracting semantics from visual scenes, and aligning visual-text cross-modal features, outputting scene semantic features related to the scene and task.

[0037] Step 102: Encode the scene semantic features and historical motion state data using a world model to determine the environmental dynamic prediction results and motion prospective features; In step 102, the scene semantic features and historical motion state data generated in step 101 are input into the look-ahead simulation world model, which outputs the prediction results of dynamic changes in the environment over multiple future frames and the look-ahead motion features of the robot itself. It can be understood that the look-ahead simulation world model is used to predict dynamic changes in the environment over multiple future frames and the motion state of the robot itself, outputting look-ahead motion features to provide future information support for trajectory planning.

[0038] Step 103: Update the full-dimensional constraints according to the environmental dynamic prediction results, and input the updated full-dimensional constraints into the iterative denoising process. The full-dimensional constraints include obstacle avoidance constraint boundaries, human-machine safety distance constraints, ZMP stability constraint boundaries and / or dynamic torque constraint weights. Step 104: Extract features from the scene semantic features, motion prospective features, environmental dynamic prediction results and historical motion data through the topological graph neural network unit to obtain joint coupling features at different times; extract features from the joint coupling features and scene semantic features through the spatiotemporal cross attention unit to obtain spatiotemporal global features and single joint local features. In step 104, the historical trajectory sequence, the scene semantic features, the target task features, and the motion prospective features are input into the trained topological graph neural network unit, which outputs joint coupling features at different times. The joint coupling features and the scene semantic features are then input into a spatiotemporal cross-attention unit, which outputs spatiotemporal global features and single-joint local motion features. The spatiotemporal cross-attention unit extracts coupling features between different joints and between different time steps based on a multi-attention mechanism, ultimately outputting spatiotemporal global features and single-joint local features.

[0039] It is understood that the topological graph neural network unit aggregates information through a topological association graph, which is constructed based on the connection relationship of robot joints, with joints as nodes of the graph and the coupling relationship between adjacent joints modeled as edges.

[0040] Step 105: Generate trajectory prior information based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features; generate an initial trajectory sequence based on the trajectory prior information; iteratively denoise the initial trajectory sequence according to the initial trajectory sequence, the noise time step, the spatiotemporal global features, the single joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory; map the fully free-degree-of-freedom predicted trajectory to joint control commands, which are used to drive joints to perform actions.

[0041] Steps 101 to 105 as illustrated in this embodiment involve responding to a received task instruction by parsing the currently collected multi-source perception data using a multimodal model to determine scene semantic features and target task features. A world model is used to encode the scene semantic features and historical motion state data to determine the environmental dynamic prediction result and motion prospective features. The full-dimensional constraints are updated based on the environmental dynamic prediction result, and the updated full-dimensional constraints are input into an iterative denoising process. These full-dimensional constraints include obstacle avoidance constraint boundaries, human-machine safety distance constraints, ZMP stability constraint boundaries, and / or dynamic torque constraint weights. A topological graph neural network unit is used to extract features from the scene semantic features, the motion prospective features, the environmental dynamic prediction result, and the historical motion data to obtain joint coupling features at different times. The spatiotemporal cross-attention unit extracts features from the joint coupling features and the scene semantic features to obtain spatiotemporal global features and single-joint local features. Based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features, trajectory prior information is generated. An initial trajectory sequence is generated based on the trajectory prior information. The initial trajectory sequence is subjected to noise sampling and iterative denoising under full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory. The fully free-degree-of-freedom predicted trajectory is mapped to joint control commands. This scheme explicitly models the kinematic coupling relationship between joints as joint coupling features through a topological graph neural network unit and uses them as input when predicting the trajectory sequence. This can reduce the risk of small errors in a single joint spreading to the whole body through the coupling relationship and realize long-term motion planning for humanoid robots.

[0042] See Figure 2 In some embodiments of step 103, step 103 includes, but is not limited to: Step 201: When the environmental dynamic prediction result represents a change in the obstacle position, update the obstacle avoidance constraint boundary. Step 202: When the environmental dynamic prediction result represents a change in the distance between the human and the robot, update the human-robot safe distance constraint; Step 203: When the environmental dynamic prediction result characterizes a change in the support region or center of gravity, update the ZMP stability constraint boundary. Step 204: When the environmental dynamic prediction result characterizes load or speed changes, update the dynamic torque constraint weights.

[0043] In this embodiment, the environmental dynamic prediction results of the future frame output by the prospective simulation world model are synchronously updated with full-dimensional constraints through steps 201 to 204, which are used to finally generate a full-degree-of-freedom predicted trajectory that satisfies the full-dimensional constraints of the current time step.

[0044] See Figure 3 In some embodiments of step 104, the historical motion data includes historical trajectory sequences and current joint state data. Step 104 includes, but is not limited to, the following: Step 301: Obtain historical trajectory sequences and current joint state data; Step 302: Input the historical trajectory sequence, the current joint state data, the scene semantic features, the environmental dynamic prediction results, and the motion prospective features into the trained topological graph neural network unit; For example, the input to the topological graph neural network unit in this embodiment is a historical trajectory sequence. Current joint status Predictive characteristics of movement Scene semantic features Environmental dynamic prediction results .

[0045] Step 303: Aggregate information of adjacent nodes from the pre-constructed topological association graph through the topological graph neural network unit to obtain joint coupling features at different times.

[0046] The topological graph uses each joint degree of freedom of the humanoid robot as a node and adjacent and coupled joints in the kinematic tree as edges. The adjacency matrix corresponds one-to-one with the kinematic tree. When performing graph convolution on the topological graph, the information of adjacent nodes is aggregated only along the adjacency matrix to explicitly model the joint coupling features.

[0047] In step 303, a feature extraction unit is constructed using a multi-layer graph convolutional network, with the number of layers ranging from [value range missing]. The layer extracts the historical trajectory sequence, current joint state data, scene semantic features, environmental dynamic prediction results, and motion prospective features from the topological association graph through a graph convolutional network to obtain joint coupling features at different times. Preferably, to address the problem of insufficient joint coupling feature extraction, the number of graph convolutional layers can be increased, or residual connections can be added to the graph convolution to avoid gradient vanishing.

[0048] For example, the formula for updating the node features of a topological graph neural network unit is:

[0049] in, For the first The output of the layer, It is a non-linear activation function. For the first The node feature matrix of the layer graph convolution, where each row of the matrix corresponds to a feature vector representing one joint degree of freedom. To add a self-loop adjacency matrix, Let be the adjacency matrix of the joint topology graph. for The corresponding degree matrix, and The first Trainable weight matrix and bias vector for layered graph convolution. The number of layers in the graph convolution.

[0050] In practical applications, humanoid robots come in various models to cope with different types of work tasks. The main difference between these models lies in the hand components. For humanoid robots with different degrees of freedom, the number of joints varies. To adapt to the new model, the denoising network needs to be redesigned and trained to accommodate the new number of joints.

[0051] This embodiment provides a method for constructing topological graph neural network units and topological association graphs, which is compatible with humanoid robot models with different degrees of freedom.

[0052] See Figure 4 In some embodiments, the topology graph neural network unit is constructed through the following steps: Step 401: Obtain the degree-of-freedom coefficients and joint connection relationships of the humanoid robot; In the field of humanoid robots, the total degrees of freedom of the humanoid robot are denoted as... ,satisfy:

[0053] in, For the head's degrees of freedom coefficients, For the waist degree of freedom coefficient, For the single leg degree of freedom coefficient, For the single-arm degree of freedom coefficients, These are the hand degrees of freedom coefficients. All are non-negative integers, and the total degrees of freedom .

[0054] Step 402: Construct a kinematic tree structure based on the joint connection relationships; In step 402, refer to Figure 5 , Figure 5 A schematic diagram of a kinematic tree structure is shown.

[0055] Step 403: Determine the number of nodes in the topological graph neural network unit based on the degree of freedom coefficients, and map the edge connection relationships between the nodes and adjacency matrices of the topological graph neural network unit to the kinematic tree structure.

[0056] Specifically, the topological graph neural network unit is constructed with a multi-layer graph convolutional network as its core. The topological graph is a graph structure constructed with each joint degree of freedom of the humanoid robot as a graph node and the kinematic coupling relationship between the joints as an edge. The adjacency matrix is ​​the mathematical representation of the topological graph, and the two are in one-to-one correspondence. Based on the values ​​of the coefficients of each degree of freedom, the topological graph neural network unit adaptively matches the number of nodes in the topological graph with the adjacency matrix according to the kinematic tree structure of the humanoid robot. The number of nodes is completely matched with the total degree of freedom N of the humanoid robot, and the edge connection relationship of the adjacency matrix corresponds one-to-one with the kinematic coupling relationship between the joints.

[0057] In steps 401 to 403 of this embodiment, the topological association graph is constructed using each joint degree of freedom of the humanoid robot as a graph node and the kinematic coupling relationship between joints as an edge. An undirected topological association graph is constructed, and a corresponding adjacency matrix A is generated. The adjacency matrix is ​​adaptively adjusted according to the robot's degree of freedom configuration, and the number of nodes is completely matched with the robot's total degree of freedom N. In this way, the topological graph neural network unit performs graph convolution only along the adjacency matrix to aggregate the information of adjacent nodes, thereby explicitly modeling the joint coupling features, rather than the black box prediction of ordinary neural networks.

[0058] In some embodiments, the method further includes: when the degree-of-freedom coefficients of the humanoid robot change, adjusting the number of nodes in the topological graph, the dimension of the adjacency matrix, the input-output dimension of the neural network unit of the topological graph, and the dimension of the fully free-degree-of-freedom predicted trajectory according to the degree-of-freedom coefficients.

[0059] For humanoid robots with different degrees of freedom, the coupling weights between joints can be recalibrated through forward kinematics, the non-zero elements of the adjacency matrix A can be updated, and the dimensions of the weight matrices of each layer of the graph convolutional network can be adjusted accordingly. This enables the adaptive adjustment of the structure of the neural network units in the topological graph, thus achieving universal adaptation for humanoid robots with different degrees of freedom.

[0060] In some embodiments, the spatiotemporal cross-attention unit includes a temporal attention branch, a spatial attention branch, a cross-modal attention branch, and a single-joint feature decoupling branch. This application also provides a method for determining spatiotemporal global features and single-joint local features based on an attention mechanism.

[0061] See Figure 6 The step of extracting features from the joint coupling features and the scene semantic features using a spatiotemporal cross-attention unit to obtain spatiotemporal global features and single-joint local features includes: Step 601: Extract the long-term temporal dependencies of the same joint at different time steps through the temporal attention branch to determine the temporal attention features; Step 602: Extract the coupling correlation between different joints at the same moment through the spatial attention branch to determine the spatial attention features; Step 603: Extract the global association between the scene semantic features and the joint motion features through the cross-modal attention branch to determine the cross-modal semantic features; Step 604: The temporal attention features, spatial attention features, and determined cross-modal semantic features are concatenated and mapped through a linear layer to obtain the spatiotemporal global features.

[0062] Step 605: Through the single-joint feature decoupling branch, extract the independent temporal motion features of each joint from the temporal attention features, and obtain the single-joint local features after linear layer mapping.

[0063] It is understandable that the dimension of the local features of a single joint is matched one-to-one with the total degrees of freedom of the robot, and is used for the generation and constraint optimization of subsequent single-joint trajectory sequences. The temporal attention branch is used to model the long-term temporal dependence of the same joint at different time steps, the spatial attention branch models the coupling relationship between different joints at the same moment, and the cross-modal attention branch models the global relationship between scene semantic features and joint motion features. The three features output from steps 101 to 103 and related steps are concatenated and mapped through a linear layer to obtain the spatiotemporal global features.

[0064] In the above embodiments, trajectory prior information is generated based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features. An initial trajectory sequence is then generated based on this trajectory prior information. This initial trajectory sequence is a temporal motion sequence of the humanoid robot's entire joint space. The final fully free-degree-of-freedom predicted trajectory generated in this application is a final temporal sequence optimized using a constraint embedding diffusion model based on this initial sequence, satisfying all dynamic, stability, and safety constraints. The final temporal sequence is the core control target sequence directly output from motion planning and can be directly sent to the joint motors for execution.

[0065] See Figure 7 In some embodiments, the iterative denoising of the initial trajectory sequence based on the initial trajectory sequence, noise time steps, the spatiotemporal global features, the single-joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory includes, but is not limited to, the following: Step 701: Input the initial trajectory sequence, noise time step, spatiotemporal global features, and single joint local features into a pre-trained joint trajectory denoising network, and output the denoised trajectory increment; Step 702: The pre-trained constraint embedding layer performs hard constraint projection and soft constraint weighted correction on the denoised trajectory increment to generate a full-degree-of-freedom predicted trajectory.

[0066] During the training phase, the constraint embedding layer performs hard constraint projection and soft constraint loss weighting on the denoised trajectory sequence.

[0067] In this embodiment, the joint trajectory denoising network is pre-trained using massive historical compliant motion trajectory data of a humanoid robot. The pre-training process includes a forward denoising process and a backward denoising process. During the inference phase, the joint trajectory denoising network performs backward denoising on the initial trajectory sequence, and after multiple iterations, obtains a fully free-degree-of-freedom predicted trajectory.

[0068] For example, the forward noise addition process is as follows: Gaussian noise is gradually added to the original compliant trajectory data, the first... The formula for the trajectory data after adding noise is:

[0069] in, The original, noise-free, fully free-degree-of-freedom trajectory data, with dimensions equal to the total degrees of freedom of the humanoid robot. match; For the first Trajectory data after adding noise; This represents the cumulative attenuation coefficient of the forward diffusion process. , For the first The noise scheduling coefficient of the step, and its range of values. ; For the first The standard Gaussian noise added step by step, It is a normal distribution function; , For the time step variable of the diffusion process.

[0070] The reverse denoising process is as follows: a noise prediction network is constructed based on the U-Net architecture, and the input is the noise-added trajectory data. Time step Using spatiotemporal global features and trajectory prior information, predict the added noise. By training the network based on the mean square error loss of predicted noise and real noise, a pre-trained joint trajectory denoising network is obtained.

[0071] Existing diffusion models can only generate geometric path trajectories, failing to incorporate the hardware physical limits, dynamic constraints, and bipedal stability constraints of humanoid robots into the generation process. The generated trajectories often suffer from joint over-limit issues, motor overload, and tipping risks, requiring secondary optimization before execution. However, this secondary optimization can cause the trajectory to deviate from the original planning objective. This application generates a trajectory sequence driving multiple degrees of freedom joints of a humanoid robot by applying hard and soft constraints during iterative denoising. This trajectory sequence satisfies the multi-degree-of-freedom control requirements of the humanoid robot and can be directly mapped to joint control commands.

[0072] See Figure 8 In some embodiments, a pre-trained constraint embedding layer performs hard constraint projection and soft constraint weighted correction on the denoised trajectory increment to generate a fully free-degree-of-freedom predicted trajectory, including but not limited to: Step 801: After each iteration, the noise trajectory output by the joint trajectory denoising network is clipped based on hard constraint projection. Specifically, hard constraint projection is used to project out-of-bounds joint angles, collision trajectories, or ZMP out-of-bounds trajectories back into the feasible region; soft constraint loss is used to impose a weighted penalty on trajectories approaching the safety boundary, torque boundary, or comfortable motion boundary, and the penalty term is applied inversely to the denoising gradient.

[0073] Step 802: Based on the gradient direction of the noise prediction with soft constraint weighting, the predicted trajectory of the full degrees of freedom is obtained.

[0074] Specifically, the soft constraint loss is used to impose a weighted penalty on trajectories that approach the safety boundary, torque boundary, or comfortable motion boundary, and the penalty term is applied inversely to the denoising gradient.

[0075] The soft constraint weighting includes dynamic torque constraints, ZMP stability constraints, obstacle avoidance constraints, and human-machine safe distance constraints.

[0076] Specifically, in step 802, constraints are defined using the multi-task total loss function formula, which is: , in, The base loss for the mean square error of noise prediction; For joint dynamic torque constraint loss; For ZMP bipedal stability constraint loss; Losses due to obstacle avoidance and human-machine safety distance constraints; This refers to the end contact force constraint loss; These are the weighting coefficients corresponding to each constraint term.

[0077] For example, the mathematical formula for the basic loss of the mean square error of noise prediction is:

[0078] in, This represents the total number of diffusion steps in the diffusion model, ranging from 200 to 1000; in this embodiment, it is set to 500. The total degrees of freedom of the humanoid robot is 29 in this embodiment, but can be adaptively adjusted according to the robot's hardware configuration. This is the time step index for the diffusion process, with a value range of [value range missing]. =1,2,..., ; This is the index for the robot's joint degrees of freedom, with a value range of... =1,2,..., ; For the first Time step, first Each joint has real Gaussian noise added, which follows a standard normal distribution; The output of the noise prediction network is the first Time step, first Predicted noise for each joint; It is the square of the L2 norm, i.e., the mean square error calculation operator.

[0079] The noise prediction mean square error base loss is the core training base loss of the diffusion model. It is used to constrain the noise prediction accuracy of the noise prediction network, is the core guarantee of trajectory generation effect, and is also the base term of all constraint losses.

[0080] Joint dynamic torque constraint loss This constraint is used to ensure that the output torque of each joint does not exceed the rated torque limit of the servo motor during the trajectory generation process, thus avoiding motor overload and trajectory failure. It is the core dynamic constraint that ensures the generated trajectory can be implemented.

[0081] For example, the formula for calculating the joint dynamic torque constraint loss is as follows:

[0082] in, This represents the total number of diffusion steps in the diffusion model, consistent with the basic loss term. The total degrees of freedom of a humanoid robot; For the time step index of the diffusion process, ; This is an index for the degrees of freedom of the joints. ; For the first Time step, first The predicted output torque of each joint is obtained by solving the predicted trajectory through the rigid body inverse dynamics model; For the first The rated maximum torque of each joint servo motor is determined by the robot's hardware parameters; This is the hinge activation function, which incurs a loss only when the joint torque exceeds the rated limit, and the constraint torque is within the hardware feasible range.

[0083] ZMP bipedal stability constraint loss The zero-moment point (ZMP), used to constrain the humanoid robot's movement throughout the entire process, always lies within the contact polygon between the two feet and the support surface, preventing the robot from tipping over during movement. It is the core stability constraint for the humanoid robot's bipedal movement.

[0084] The formula for calculating the constraint loss of ZMP bipedal stability is as follows:

[0085] in, This represents the total number of diffusion steps in the diffusion model, consistent with the previous statement; For the time step index of the diffusion process, ; , For the first At the time step, the x and y coordinates of the robot's zero-torque point on the horizontal plane of the world coordinate system; , For the first Time step, the maximum and minimum boundaries of the bipedal supported polygon in the x-axis direction; , For the first The time step is the maximum and minimum boundaries of the bipedal support polygon in the y-axis direction; loss only occurs when the ZMP coordinates exceed the boundaries of the support polygon, forcing the ZMP to always remain within the stable support area to ensure that the bipedal movement does not tip over.

[0086] Obstacle avoidance and human-machine safety distance constraint loss The minimum distance between the robot body and environmental obstacles and human beings is always greater than a safety threshold to avoid collisions and ensure operational safety in dynamic scenarios. The formula for calculating the loss due to obstacle avoidance and human-robot safety distance constraint is as follows:

[0087] in, This represents the total number of diffusion steps in the diffusion model, consistent with the previous statement; This refers to the number of critical collision detection points on the robot body, typically taken from easily collided locations such as joints and link ends. For the time step index of the diffusion process, ; This serves as an index for collision detection points on the robot body. ; The preset safe distance threshold is typically 0.05m for industrial scenarios and 0.3m for home human-computer interaction scenarios; For the first Time step, first The Euclidean distance between each collision detection point and the nearest obstacle / human body.

[0088] Loss occurs only when the distance between the detection point and the obstacle is less than the safety threshold, and the forced trajectory meets the requirements for obstacle avoidance and human-machine safety.

[0089] End contact force constraint loss It is used to constrain the contact force between the end effector and the work object, so as to avoid damage to the workpiece in industrial assembly and the operated items in household scenarios, and to ensure the accuracy and safety of the operation.

[0090] For example, the formula for calculating the end contact force constraint loss is as follows:

[0091] in, The contact-end point prediction submodule outputs the preset contact time interval between the end point and the work object. This is the index of the time steps within the contact time interval; For the first The resultant force of contact force detected by the six-dimensional force sensor of the time-step end effector; The upper limit threshold of the contact force is determined by the work scenario (e.g., 5N for precision assembly, 20N for household gripping). This is the lower limit threshold of the contact force, ensuring effective contact between the end effector and the workpiece during operation.

[0092] The end contact force constraint loss simultaneously constrains the upper and lower limits of the contact force, thus preventing both excessive contact force from damaging the workpiece and insufficient contact force from causing operational failure.

[0093] After each denoising iteration in step 801, the feasible region of the predicted trajectory output by the noise prediction network is clipped. Trajectory values ​​that exceed the joint angle limit, angular velocity limit, and angular acceleration limit are mapped to the upper and lower limit threshold ranges of the corresponding parameters to ensure that the trajectory does not exceed the physical limits of the hardware.

[0094] In the noise prediction network inference process in step 802, loss terms for dynamic torque constraint, ZMP stability constraint, obstacle avoidance constraint, and human-machine safety distance constraint are added to the basic mean square error loss function. Corresponding weight coefficients are set for each constraint term. The gradient direction of noise prediction is corrected through multi-task loss so that the generated trajectory meets the requirements of safety and stability in all dimensions.

[0095] This embodiment designs a constraint-embedded diffusion model that directly embeds the hard constraints of the humanoid robot, such as kinematics, dynamics, bipedal stability, and scene safety, into each step of the denoising process of the diffusion model. Through a dual mechanism of hard constraint projection and soft constraint loss weighting, it ensures that the generated trajectory meets the robot's hardware limits and scene safety requirements, thus solving the problem of unexecutable generated trajectories from the root.

[0096] In some embodiments, the trajectory prior information includes endpoint location constraints and full-time contact state constraints; generating trajectory prior information based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features includes: The spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features are input into a pre-trained contact endpoint prediction module, which outputs the endpoint position constraints and the contact state constraints. The contact endpoint prediction module is constructed based on a bidirectional gated recurrent unit (GRU) and a constrained fully connected layer.

[0097] In this embodiment, a contact-endpoint dual-guided trajectory generation mechanism is proposed. The endpoint of the operation and the contact state throughout the entire time sequence are predicted simultaneously. The constraints of stable contact of both feet and end-effector contact are injected into the trajectory generation process in advance, providing global guidance for diffusion generation, avoiding the accumulation of errors in autoregressive stepwise prediction, and ensuring the stability of the humanoid robot's movement throughout the entire process.

[0098] The following section provides a detailed introduction and explanation of the solutions in this embodiment of the invention, using specific model training and inference scenarios: This application provides an implementation step for the model pre-training stage: Step 1: Construction and pre-training of the multimodal semantic parsing and world-leading simulation module.

[0099] The multimodal semantic parsing and world-looking simulation module includes a multi-source sensing data preprocessing submodule, a VLA multimodal semantic parsing module, and a world-looking simulation model submodule.

[0100] The multi-source sensing data preprocessing submodule is used to complete hardware time synchronization, denoising, and standardization of multi-source sensor data, providing compliant and standardized input data for all subsequent modules. It collects multi-source sensing data synchronized with hardware timestamps from the humanoid robot, including RGB-D visual data, angle / angular velocity / angular acceleration data from the whole-body joint encoder, six-dimensional force sensor data from the foot, force sensing data from the end effector, and posture data from the torso IMU. The module performs data denoising, normalization, and time-series alignment to construct the model training dataset. For different industrial and household application scenarios, it can expand the dataset with scenario-specific sensing data subsets to improve the model's scenario adaptability.

[0101] The input parameters for the multi-source sensing data preprocessing submodule are:

[0102] In the formula, The timestamp index for sensor data, and the dimensions and physical meanings of each sub-parameter are as follows: : RGB-D visual data at any given time. , 4 represents the image height and width, and 4 represents the RGB three-channel + depth channel; : Real-time data on the angles of all joints throughout the body. This embodiment represents the total degrees of freedom of the robot. ; : Real-time angular velocity data of all joints; : Real-time data on the angular acceleration of all joints throughout the body; : Real-time bipedal six-dimensional force sensor data: 6-dimensional force / torque for a single foot, totaling 12 dimensions for both feet; : Real-time data from the six-dimensional force sensor of the dual-arm end effector: 6-dimensional force / torque for a single arm, and a total of 12 dimensions for both arms; : Real-time torso IMU attitude data, 3D attitude angle + 3D angular velocity.

[0103] The output parameters of the multi-source sensing data preprocessing submodule are: Preprocessed standardized time series dataset:

[0104] All data underwent time-series alignment, outlier removal, and min-max normalization. The core normalization formula is:

[0105] In the formula: For raw sensor data, , These are the physical limits of the corresponding sensor data, including the minimum and maximum values. For the normalized data, a unified mapping is performed. Interval.

[0106] The VLA multimodal semantic parsing submodule is used to complete natural language task instruction parsing, visual scene semantic extraction, visual-text cross-modal feature alignment, and output global semantic features of the scene and task.

[0107] The input parameters for the VLA multimodal semantic parsing submodule are preprocessed visual data. A sequence of natural language job instructions input by the user. ,in The length of the text sequence. Text token; scene semantic features : Coverage of trajectory planning across the entire time step Global semantic features of the visual scene As a feature dimension, this embodiment and target task characteristics : Global feature vectors of task objectives and constraints mapped by natural language job instructions.

[0108] The VLA multimodal semantic parsing submodule uses contrastive learning loss to achieve visual-text cross-modal feature alignment. The training loss formula is as follows:

[0109] In the formula: To train batch size, The cosine similarity function is used. For the first Visual features of each sample For the first Text features of each sample This is the temperature coefficient, with a value ranging from 0.05 to 0.2.

[0110] The forward-looking simulation world model (Transformer architecture) is used to predict the dynamic changes of the environment and the motion state of the robot body in the future multiple frames, output forward-looking simulation features, provide future information support for trajectory planning, and avoid the accumulation of long-term planning errors.

[0111] The input parameters for the prospective simulation world model are historical data. Motion state sequence of frames and scene semantic features , , The length of the historical frame, with a value ranging from 5 to 20.

[0112] The output of the look-ahead simulation world model is the look-ahead motion characteristics of the robot body. :future Features for predicting robot motion states in frames. The number of frames to be predicted ranges from 10 to 50; environmental dynamic prediction results. :future Intra-frame The predicted trajectory coordinates of one dynamic obstacle, and 2 are the horizontal x and y axis coordinates.

[0113] The look-ahead simulation world model is pre-trained using an autoregressive prediction loss. The training loss formula is as follows:

[0114] In the formula: The future predicted by the model Frame joint angle, For the future of real-world data collection Frame joint angle; The coordinates of the obstacle predicted by the model. These are the coordinates of the actual obstacles.

[0115] Step 2: Construction of the Joint-Coupled Spatiotemporal Feature Encoding Module Step 2.1: Construction of Topological Association Graph: Using each joint degree of freedom of the humanoid robot as a graph node, and the joint connection relationships and kinematic coupling relationships of the humanoid robot's kinematic tree structure as edges, construct an undirected topological association graph and generate the corresponding adjacency matrix. The adjacency matrix can be adaptively adjusted according to the robot's degrees of freedom configuration, with the number of nodes corresponding to the robot's total degrees of freedom. Perfect matching; for humanoid robots with different degrees of freedom, the coupling weights between joints can be recalibrated using forward kinematics, and the adjacency matrix can be updated. The non-zero element values.

[0116] Step 2.2: Construction of Topological Graph Neural Network Units: Feature extraction units are constructed using multi-layer graph convolutional networks, with the number of layers ranging from [value range missing]. The formula for updating the features of a layer and its nodes is: .

[0117] Graph convolutional networks can be used to extract historical trajectory data, current joint state data, motion prospective features, and joint coupling features at different times in the environmental dynamic prediction results. To address the problem of insufficient extraction of joint coupling features, the number of graph convolutional layers can be increased, or residual connections can be added to the graph convolution to avoid gradient vanishing.

[0118] Step 2.3: Construction of Spatiotemporal Cross-Attention Units: Parallel temporal attention branches, spatial attention branches, and cross-modal attention branches are set up. The temporal attention branch adopts a multi-head self-attention mechanism to model the long-term temporal dependencies of the same joint at different time steps; the spatial attention branch adopts a graph attention mechanism to model the coupling relationship between different joints at the same moment; the cross-modal attention branch adopts a cross-attention mechanism to model the global relationship between scene semantic features and joint motion features. The output features of the three branches are concatenated and mapped through a linear layer to obtain spatiotemporal global features and single-joint local motion features. To address the problem of insufficient modeling of long temporal dependencies, the number of attention heads can be increased, or a sliding window mechanism can be added to the temporal attention branch to reduce the amount of computation while improving the ability to extract long temporal features.

[0119] Step 3: Construction and training of the contact-endpoint prediction submodule. A contact-endpoint prediction submodule is constructed based on a bidirectional gated recurrent unit (GRU) and a constrained fully connected layer. The hidden layer dimension ranges from 128 to 512. The inputs are spatiotemporal global features, scene semantic features, and target task features. The outputs are the coordinates of the target endpoint, stable contact point parameters between the robot and the environment, and the contact state time series. These are integrated to obtain trajectory prior information with both endpoint and contact constraints. The stable contact point parameters and contact state time series include the contact area between the two feet and the support surface, the contact force threshold, the ZMP feasible interval for the zero-moment point, and the contact time series, contact posture, and contact force constraints between the end effector and the target object. Based on smoothing... The loss function uses a paired dataset of "trajectory-endpoint-contact state" from a humanoid robot's work scenario to complete model training, enabling the model to simultaneously output endpoint position constraints and full-time contact state constraints. To address the issues of large endpoint prediction bias and contact state prediction not meeting the bipedal stability requirements, ZMP stability constraints and endpoint position weight coefficients can be added to the loss function, or the hidden layer dimension of the bidirectional gated recurrent unit can be increased to improve the temporal feature extraction capability.

[0120] Specifically, the trajectory prior information is not a simple "data splicing," but rather a standardized end-to-end processing of the multi-dimensional constraint information output by the sub-modules, involving "temporal dimension alignment → feature fusion encoding → constraint mask generation → structured encapsulation," ultimately yielding structured prior information that can be directly input into the diffusion model. The specific integration steps are as follows: 3.1 Time-series dimension unified alignment The endpoint coordinates and stable contact point parameters output by the submodule are mapped to a time series dimension of the same length as the trajectory planning through linear interpolation, so that the endpoint constraints and contact point constraints cover the entire time step of trajectory generation. This ensures that all constraint information is fully matched with the temporal dimension of the trajectory sequence.

[0121] The alignment formula is:

[0122]

[0123] In the formula: It is a linear interpolation function. The first time after timing alignment Time step endpoint coordinates The first time after timing alignment Time step contact point parameters.

[0124] 3.2 Multi-constraint Feature Fusion Coding The aligned endpoint coordinate sequence, stable contact point parameter sequence, and contact state time sequence are concatenated and fused along the feature dimension to obtain the fused constraint features. The fusion formula is:

[0125] In the formula: For the first Contact state sequence at time step, This is the feature splicing function.

[0126] 3.3 Contact State Constraint Mask Generation Based on the contact state time sequence, a constraint mask matrix for contact / non-contact states is generated. During the contact phase, the mask value is 1 to retain contact force and contact posture constraints; during the non-contact phase, the mask value is 0 to mask invalid constraints and avoid interference from contact force constraints in the non-contact phase to generate the trajectory.

[0127] 3.4 Structured Encapsulation of Trajectory Prior Information Fusion constraint features With constraint mask matrix By integrating the data, we obtain prior trajectory information with both endpoint and contact constraints. Its dimensions perfectly match the input dimensions of the diffusion model, and it can be directly input into the subsequent constraint embedding diffusion generation submodule.

[0128] This application does not directly splice the original output data, but completes time alignment, constraint masking, and dimension matching to ensure that the prior information corresponds one-to-one with the full time step and full degrees of freedom of the trajectory generation, thereby avoiding the trajectory generation effect of the diffusion model from the root cause of invalid constraints interfering with it.

[0129] Step 4: Construction and pre-training of the constrained embedding diffusion generation submodule This module includes a pre-trained joint trajectory denoising network, a constraint embedding layer, and a trajectory initialization unit. The specific construction and pre-training steps are as follows: Joint trajectory denoising network pre-training: Collect a dataset of historical compliant motion trajectories of a humanoid robot, including full-scene motion trajectories such as walking, grasping, and working, and complete forward denoising and backward denoising training: Forward noise addition process: Set the total number of diffusion steps The value ranges from 200 to 1000, and the data is derived from the original noiseless full-degree-of-freedom trajectory data. Gaussian noise is added gradually, the first... The trajectory data after adding noise is ; Reverse denoising process: A noise prediction network is built based on the U-Net architecture, and the input is the noisy trajectory data. Time step Using spatiotemporal global features and trajectory prior information, predict the added noise. The network is trained based on the mean square error loss of predicted noise and real noise, and a pre-trained joint trajectory denoising network is obtained after convergence. To address the issues of non-convergence during diffusion model training and large noise prediction bias, the noise scheduling coefficient can be adjusted. The range of values ​​can be adjusted to reduce the noise rate; or cross-layer connections of spatiotemporal global features can be added to U-Net to strengthen the guiding role of constraint information in noise prediction; to address the problem of insufficient inference speed, the DDIM denoising diffusion implicit model can be used to accelerate sampling and reduce the number of inference sampling steps.

[0130] Constraint embedding layer design: In each iteration of diffusion denoising, hard constraint projection and soft constraint loss weighting are performed simultaneously, and the total loss function for multiple tasks is as shown above.

[0131] Hard constraint projection: After each denoising iteration, the feasible region of the predicted trajectory output by the noise prediction network is clipped. Trajectory values ​​that exceed the joint angle limit, angular velocity limit, and angular acceleration limit are mapped to the upper and lower limit threshold ranges of the corresponding parameters to ensure that the trajectory does not exceed the hardware physical limit. Soft constraint loss weighting: In the training and inference process of the noise prediction network, loss terms of dynamic torque constraint, ZMP stability constraint, obstacle avoidance constraint, and human-machine safety distance constraint are added to the basic mean square error loss function. Corresponding weight coefficients are set for each constraint term. The gradient direction of noise prediction is corrected through multi-task loss so that the generated trajectory meets the requirements of safety and stability in all dimensions. To address the trajectory generation failure caused by conflicts among multiple constraints, the weight coefficients of each constraint can be adjusted according to the scenario priority. For example, in industrial precision operation scenarios, the weight of the end contact force constraint can be increased, and in home scenarios, the weight of the human-machine safety distance constraint can be increased. To address the trajectory jitter problem caused by hard constraint projection, a constraint smoothing term can be added in the denoising iteration to ensure the continuity of the trajectory.

[0132] Trajectory initialization unit construction: The trajectory initialization unit is constructed based on multilayer perceptron and topological graph neural network. The input is trajectory prior information, spatiotemporal global features, and single-joint local motion features. The output is an initial trajectory sequence that conforms to the robot's current motion state and satisfies the endpoint and initial contact constraints. To address the problem of the initial trajectory deviating from the target, an endpoint position constraint term can be added to the loss function of the initialization unit to strengthen the guiding role of trajectory prior information.

[0133] Step 5: Construction of the Predictive-Control End-to-End Closed-Loop Module The built-in rigid body inverse dynamics solver and PID proportional-integral-derivative feedback compensation unit can map the predicted trajectory of the full degrees of freedom into joint torque control commands and position loop control commands, with a control frequency of no less than 100Hz. At the same time, a data feedback link is set up to transmit the actual joint state data, force data, and visual perception data to the front-end module in real time, so as to realize dynamic trajectory correction and replanning. For different models of servo actuators, the PID parameters and the inertia parameters of the inverse dynamics solution can be adjusted accordingly to adapt to the control characteristics of different hardware.

[0134] This application provides a deployment and implementation step for a motion planning method. All modules pre-trained in the above embodiments are deployed in the main control computing unit of the humanoid robot. When the humanoid robot is working on-site, it performs whole-body motion planning and closed-loop control according to the following steps.

[0135] Step (1), real-time acquisition and preprocessing of multi-source sensing data: Multi-source sensing data of the humanoid robot is collected synchronously through the multi-source sensing unit. After noise reduction, timestamp alignment and normalization preprocessing are completed, the data is transmitted to the main control computing unit. At the same time, natural language operation instructions input by the user are received.

[0136] Step (2) Multimodal semantic parsing and world-looking simulation: Input the preprocessed visual data and natural language task instructions into the VLA multimodal large model, and output the scene semantic features and target task features; input the scene semantic features and historical motion state data into the world-looking simulation model, and output the future multi-frame environment dynamic prediction results and robot body motion looking-ahead features.

[0137] Step (3) Joint coupling spatiotemporal feature encoding: After temporal position encoding of historical full-degree-of-freedom motion trajectory data, current joint state data, and motion prospective features, input them into the topology graph neural network unit to extract joint coupling features at different times; input the joint coupling features and scene semantic features into the spatiotemporal cross-attention unit to output spatiotemporal global features and single joint local motion features.

[0138] Step (4) Contact-Endpoint Dual Constraint Prediction: Input the spatiotemporal global features, scene semantic features, and target task features into the pre-trained contact-endpoint prediction submodule, and output the trajectory prior information with both endpoint and contact constraints.

[0139] Step (5) Constraint Embedding Diffusion Trajectory Generation: Input the trajectory prior information, spatiotemporal global features, and single-joint local motion features into the trajectory initialization unit to generate the initial trajectory sequence; input the initial trajectory sequence, spatiotemporal global features, single-joint local motion features, and trajectory prior information into the joint trajectory denoising network simultaneously, use DDIM for fast sampling, and the number of sampling steps ranges from 20 to 100. In each denoising process, inject full-dimensional constraints through the constraint embedding layer to finally generate a full-degree-of-freedom predicted trajectory that satisfies all constraints.

[0140] Step (6) Prediction-control closed-loop execution: The generated full-degree-of-freedom predicted trajectory is input into the prediction-control end-to-end closed-loop module and mapped to joint control commands. The actuator unit is driven by the real-time motion control unit to complete the corresponding action. At the same time, the actual motion state data and real-time perception data are sent back to the world-leading simulation module and the joint coupling spatiotemporal feature encoding module. When the environment or work target changes dynamically, the trajectory is triggered to replan and correct in real time, thus completing the full-process closed-loop control.

[0141] This application also provides a complete example of adapting and modifying a pre-trained model for a humanoid robot with different degrees of freedom.

[0142] Application scenario: The target deployment is a 17-DOF (DoF) home service humanoid robot, with the following DDO configuration: head DDO coefficient. waist degree of freedom coefficient Single leg degree of freedom coefficient Total degrees of freedom for both legs: 6; Degrees of freedom coefficient for one arm: [not specified] Total degrees of freedom of both arms: 8; Hand degrees of freedom coefficient: Total degrees of freedom of the whole body Unlike the original 29-DOF basic configuration, the pre-trained model needs to be adapted and modified to match the hardware structure of the target robot.

[0143] This application provides a pre-trained model adaptation and modification step for a humanoid robot with different degrees of freedom, including: (1) Modify the topology graph: Based on the target robot's 17 degrees of freedom configuration, reconstruct the topology graph, setting the number of nodes to 17, which is consistent with the total degrees of freedom. Complete matching; based on the kinematic tree structure of the target robot, the kinematic coupling relationship between joints is recalibrated, and the adjacency matrix is ​​updated. The non-zero values ​​of the adjacency matrix ensure a one-to-one correspondence between the edge connections and the physical connections and kinematic coupling between joints; when the target robot includes a dexterous hand ( When doing so, the hand joint nodes and corresponding coupling edges are simultaneously expanded in the topological association graph.

[0144] (2) Modify the joint coupling spatiotemporal feature encoding module: Adjust the input and output dimensions of the topological graph neural network unit accordingly, change the dimension of the node feature matrix from 29×128 to 17×128, and simultaneously adjust the trainable weight matrix of each layer of graph convolution. The dimension of the attention unit is adjusted to match the number of nodes; the input dimension of the spatiotemporal cross attention unit is simultaneously adjusted to 17, while the rest of the network structure and attention mechanism logic remain unchanged.

[0145] (3) Mandatory modifications to the constraint embedding diffusion generation submodule: Adjust the input and output dimensions of the trajectory initialization unit and the joint trajectory denoising network accordingly, and change the feature dimension of the trajectory data from 29 to 17 to match the total degrees of freedom of the target robot; In the hard constraint projection module, update the angle, angular velocity, and angular acceleration limit thresholds of the 17 joints to correspond one-to-one with the hardware physical parameters of the target robot.

[0146] In this embodiment, a historical compliant motion trajectory dataset of a 17-DOF target robot is used. The initial learning rate is set to 1 / 10 of the original pre-training learning rate, and the number of training rounds is set to 1 / 5 of the original pre-training rounds. This completes the model adaptation. Without full retraining, the model can be adapted to the hardware of the target robot. The modified model can be fully adapted to the hardware structure of the 17-DOF humanoid robot, accurately model the joint coupling relationship of the target robot, and the generated trajectory fully conforms to the hardware physical limits of the target robot. There is no need to perform full pre-training again, which greatly reduces the adaptation cost of different hardware.

[0147] Please see Figure 9This application also provides a humanoid robot whole-body motion planning device that can implement the above-described method. The device includes: The first module is used to respond to the received task instructions by analyzing the currently collected multi-source perception data through a multimodal model to determine the scene semantic features and target task features; The second module is used to encode scene semantic features and historical motion state data through a world model to determine the dynamic prediction results of the environment and the prospective features of motion. The third module is used to update the full-dimensional constraints based on the environmental dynamic prediction results, and input the updated full-dimensional constraints into the iterative denoising process. The full-dimensional constraints include obstacle avoidance constraint boundaries, human-machine safety distance constraints, ZMP stability constraint boundaries and / or dynamic torque constraint weights. The fourth module is used to extract features from the scene semantic features, the motion prospect features, the environmental dynamic prediction results and historical motion data through the topological graph neural network unit to obtain joint coupling features at different times, and to extract features from the joint coupling features and the scene semantic features through the spatiotemporal cross attention unit to obtain spatiotemporal global features and single joint local features. The fifth module is used to generate trajectory prior information based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features; generate an initial trajectory sequence based on the trajectory prior information; iteratively denoise the initial trajectory sequence according to the initial trajectory sequence, the noise time step, the spatiotemporal global features, the single joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory; and map the fully free-degree-of-freedom predicted trajectory into joint control commands, which are used to drive joints to perform actions.

[0148] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0150] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the methods described in the embodiments of this application. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0152] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0153] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0154] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0155] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0156] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0157] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0158] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0161] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0162] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0164] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0166] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for planning the whole-body motion of a humanoid robot, characterized in that, The method includes the following steps: In response to receiving a task instruction, the system analyzes the currently collected multi-source perception data using a multimodal model to determine the scene semantic features and target task features. By encoding scene semantic features and historical motion state data using a world model, the dynamic prediction results of the environment and the prospective features of motion are determined. The full-dimensional constraints are updated based on the environmental dynamic prediction results, and the updated full-dimensional constraints are input into the iterative denoising process. The full-dimensional constraints include obstacle avoidance constraint boundaries, human-machine safety distance constraints, ZMP stability constraint boundaries and / or dynamic torque constraint weights. The scene semantic features, motion prospective features, environmental dynamic prediction results and historical motion data are extracted by the topological graph neural network unit to obtain joint coupling features at different times. The joint coupling features and scene semantic features are extracted by the spatiotemporal cross attention unit to obtain spatiotemporal global features and single joint local features. Based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features, trajectory prior information is generated. An initial trajectory sequence is generated based on the trajectory prior information. The initial trajectory sequence is iteratively denoised according to the initial trajectory sequence, the noise time step, the spatiotemporal global features, the single joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory. The fully free-degree-of-freedom predicted trajectory is mapped to joint control commands, which are used to drive joints to perform actions.

2. The method according to claim 1, characterized in that, The step of updating the full-dimensional constraints based on the environmental dynamic prediction results includes: When the environmental dynamic prediction result represents a change in the obstacle's position, the obstacle avoidance constraint boundary is updated. When the environmental dynamic prediction result represents a change in the distance between the human and the robot, the human-robot safe distance constraint is updated. When the environmental dynamic prediction results characterize changes in the support region or center of gravity, update the ZMP stability constraint boundary. When the environmental dynamic prediction results characterize load or speed changes, the dynamic torque constraint weights are updated.

3. The method according to claim 1, characterized in that, The historical motion data includes historical trajectory sequences and current joint state data. The topological graph neural network unit extracts features from the scene semantic features, the motion prospective features, the environmental dynamic prediction results, and the historical motion data to obtain joint coupling features at different times, including: Obtain the historical trajectory sequence and the current joint state data; The historical trajectory sequence, the current joint state data, the scene semantic features, the environmental dynamic prediction results, and the motion prospective features are input into the trained topological graph neural network unit; The joint coupling features at different times are obtained by aggregating information of adjacent nodes from the pre-constructed topological association graph through the topological graph neural network unit. The topological graph uses each joint degree of freedom of the humanoid robot as a node and adjacent and coupled joints in the kinematic tree as edges. The adjacency matrix corresponds one-to-one with the kinematic tree. When performing graph convolution on the topological graph, the information of adjacent nodes is aggregated only along the adjacency matrix to explicitly model the joint coupling features.

4. The method according to claim 3, characterized in that, The topology graph neural network unit is constructed through the following steps: Obtain the degree-of-freedom coefficients and joint connection relationships of the humanoid robot; Construct a kinematic tree structure based on the joint connection relationships; The number of nodes in the topological graph neural network unit is determined based on the degree of freedom coefficients, and the edge connection relationships between the nodes and the adjacency matrix of the topological graph neural network unit are mapped one-to-one with the kinematic tree structure.

5. The method according to claim 4, characterized in that, The method further includes: When the degree-of-freedom coefficients of the humanoid robot change, the number of nodes in the topological graph, the dimension of the adjacency matrix, the input-output dimension of the neural network unit of the topological graph, and the dimension of the fully free-degree-of-freedom predicted trajectory are adjusted according to the degree-of-freedom coefficients.

6. The method according to claim 1, characterized in that, The spatiotemporal cross-attention unit includes a temporal attention branch, a spatial attention branch, a cross-modal attention branch, and a single-joint feature decoupling branch; The method involves extracting features from the joint coupling features and the scene semantic features using a spatiotemporal cross-attention unit to obtain spatiotemporal global features and single-joint local features, including: The long-term temporal dependencies of the same joint at different time steps are extracted through the temporal attention branch to determine the temporal attention features; The spatial attention branch is used to extract the coupling correlation between different joints at the same moment to determine the spatial attention features; The cross-modal attention branch is used to extract the global association between the scene semantic features and the joint motion features to determine the cross-modal semantic features; The spatiotemporal global features are obtained by concatenating the temporal attention features, the spatial attention features, and the cross-modal semantic features and then mapping them through a linear layer. By decoupling the single-joint feature branch, the independent temporal motion features of each joint are extracted from the temporal attention features, and the single-joint local features are obtained after linear layer mapping.

7. The method according to claim 1, characterized in that, The step of iteratively denoising the initial trajectory sequence based on the initial trajectory sequence, noise time steps, the spatiotemporal global features, the single-joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory includes: The initial trajectory sequence, noise time steps, spatiotemporal global features, and single-joint local features are input into a pre-trained joint trajectory denoising network, which outputs the denoised trajectory increment. By using a pre-trained constraint embedding layer to perform hard constraint projection and soft constraint weighted correction on the denoised trajectory increment, a fully free-degree-of-freedom predicted trajectory is generated.

8. The method according to claim 7, characterized in that, The process of generating a fully free-degree-of-freedom predicted trajectory by performing hard-constraint projection and soft-constraint weighted correction on the denoised trajectory increment through a pre-trained constraint embedding layer includes: After each iteration, the noise trajectory output by the joint trajectory denoising network is clipped based on hard constraint projection. The gradient direction of the noise prediction is corrected based on soft constraints, and the predicted trajectory of the full degree of freedom is obtained. The updated full-dimensional constraints include dynamic torque constraints, ZMP stability constraints, obstacle avoidance constraints, and human-machine safety distance constraints.

9. The method according to claim 1, characterized in that, The trajectory prior information includes endpoint position constraints and full-time contact state constraints; The generation of trajectory prior information based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features includes: The spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features are input into a pre-trained contact endpoint prediction module, which outputs the endpoint position constraints and the contact state constraints. The contact endpoint prediction module is constructed based on a bidirectional gated recurrent unit (GRU) and a constrained fully connected layer.

10. A humanoid robot whole-body motion planning device, characterized in that, The device includes: The first module is used to respond to the received task instructions by analyzing the currently collected multi-source perception data through a multimodal model to determine the scene semantic features and target task features; The second module is used to encode scene semantic features and historical motion state data through a world model to determine the dynamic prediction results of the environment and the prospective features of motion. The third module is used to update the full-dimensional constraints based on the environmental dynamic prediction results, and input the updated full-dimensional constraints into the iterative denoising process. The full-dimensional constraints include obstacle avoidance constraint boundaries, human-machine safety distance constraints, ZMP stability constraint boundaries and / or dynamic torque constraint weights. The fourth module is used to extract features from the scene semantic features, the motion prospect features, the environmental dynamic prediction results and historical motion data through the topological graph neural network unit to obtain joint coupling features at different times, and to extract features from the joint coupling features and the scene semantic features through the spatiotemporal cross attention unit to obtain spatiotemporal global features and single joint local features. The fifth module is used to generate trajectory prior information based on the spatiotemporal global features, the scene semantic features, the environmental dynamic prediction results, and the target task features; generate an initial trajectory sequence based on the trajectory prior information; iteratively denoise the initial trajectory sequence according to the initial trajectory sequence, the noise time step, the spatiotemporal global features, the single joint local features, and the updated full-dimensional constraints to obtain a fully free-degree-of-freedom predicted trajectory; and map the fully free-degree-of-freedom predicted trajectory into joint control commands, which are used to drive joints to perform actions.

11. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 9.

Images