Robot whole body action control method, robot and computer program product

Abstract

The invention relates to the technical field of intelligent technology, and provides a whole-body action control method of a robot, the robot and a computer program product. The method comprises the following steps: acquiring a multi-modal feature sequence of a robot; according to the multi-modal feature sequence, sub-targets of the robot are determined, and the sub-targets are used for representing action expectation information of the robot in the current stage; and based on the sub-targets, cooperatively controlling upper limb actions and lower limb actions of the robot. By adopting the method, the actions of the upper and lower limbs of the robot can be controlled to meet the stable expected actions of the corresponding stages in each stage when the robot executes the long action link task, so that the stable coordination of the actions of the upper and lower limbs of the robot can be kept in each stage, and the instability risk is reduced.

Description

Technical Field

[0001] This application relates to the field of embodied intelligence technology, and in particular to a method for controlling the whole body motion of a robot, a robot, and a computer program product. Background Technology

[0002] With the development of embodied intelligence and large-scale modeling technologies, the whole-body motion control of robots is gradually evolving from the traditional cascaded process of perception, modeling, planning, and control to an end-to-end decision-making paradigm centered on the Vision Language Action (VLA) model. This paradigm, by unifying the representation of language intent, visual observation, and robot state, can achieve a direct mapping "from semantics to action" in complex tasks with fewer rules and lower manual modeling costs. However, when faced with embodied intelligence tasks with multi-stage, long-sequential motion chains, such as object handling or sorting, robots experience significant load fluctuations and environmental disturbances at different stages, making it difficult to maintain steady-state coordination of the robot's upper and lower limb movements and posing a high risk of instability. Summary of the Invention

[0003] In view of this, embodiments of this application provide a robot whole-body motion control method, a robot, and a computer program product, which can maintain steady-state coordination of the robot's upper and lower limb movements and reduce the risk of instability during the robot's execution of long motion chain tasks.

[0004] The first aspect of this application provides a method for controlling the whole-body motion of a robot, including:

[0005] Obtain the robot's multimodal feature sequence;

[0006] Based on the multimodal feature sequence, the robot's sub-objectives are determined. The sub-objectives are used to characterize the robot's expected action information at the current stage.

[0007] Based on sub-goals, the robot's upper and lower limb movements are controlled collaboratively.

[0008] The technical solution of this application first obtains the multimodal feature sequence of the robot, then determines the robot's sub-targets based on the multimodal feature sequence. These sub-targets characterize the robot's expected action information at the current stage. Finally, based on these sub-targets, the robot's upper limb and lower limb movements are coordinated and controlled. With this setup, the robot's upper and lower limb movements can be controlled at each stage of a long motion chain task to simultaneously meet the stable expected actions of the corresponding stage. This maintains steady-state coordination of the robot's upper and lower limb movements at each stage, reducing the risk of instability.

[0009] In one implementation of this application, the coordinated control of the robot's upper and lower limb movements based on sub-objectives includes:

[0010] Based on the sub-objectives, determine the desired pose of the robot's upper limb end effector;

[0011] By using inverse kinematics, the desired pose is converted into the desired state of the robot's upper limb joints;

[0012] Based on the current and desired states of the upper limb joints, a reference motion trajectory for the upper limb joints is generated.

[0013] Control the upper limb joints according to the reference movement trajectory.

[0014] In one implementation of this application, the coordinated control of the robot's upper and lower limb movements based on sub-objectives includes:

[0015] Based on the contact time markers contained in the sub-objectives, determine the degree of opening and closing of the robot's mechanical gripper and the expected gripping force within the corresponding time period;

[0016] The mechanical gripper is controlled according to the degree of opening and closing and the desired clamping force.

[0017] In one implementation of this application, the coordinated control of the robot's upper and lower limb movements based on sub-objectives includes:

[0018] Based on the sub-objectives and obstacle information in the robot's environment, the expected footprint sequence and gait phase of the robot are generated;

[0019] Based on the desired footprint sequence and gait phase, generate the desired zero-moment point trajectory that always lies within the supporting polygon;

[0020] Transform the desired zero-moment point trajectory into the desired centroid trajectory;

[0021] The robot's lower limb movements are controlled according to the desired centroid trajectory, so that the robot's actual centroid trajectory follows the desired centroid trajectory.

[0022] In one implementation of this application, controlling the robot's lower limb movements according to a desired centroid trajectory includes:

[0023] Construct objective functions to minimize the errors between the robot's actual footprint sequence and the desired footprint sequence, the errors between the actual centroid trajectory and the desired centroid trajectory, and the robot's lower limb joint torques;

[0024] Construct constraints for the robot's foot parameters, friction cone parameters, and lower limb joint parameters;

[0025] Solve for the control parameters of the robot's lower limb joints while satisfying the constraints and minimizing the objective function;

[0026] The lower limb joints are controlled according to the control parameters.

[0027] In one implementation of this application, the method further includes:

[0028] Detect the external forces and torques acting on the robot's upper limb extremities;

[0029] Based on external forces and external torques, correct the desired footprint sequence and the desired centroid trajectory.

[0030] In one implementation of this application, determining the robot's sub-targets based on a multimodal feature sequence includes:

[0031] The multimodal feature sequence is input into a trained temporal fusion neural network to perform feature fusion processing on the time dimension, and the latent vectors of each time step are obtained.

[0032] The sub-objectives are obtained by regressing the latent vectors at each time step using the sub-objective regression head.

[0033] In one implementation of this application, the method further includes:

[0034] The hidden vectors at each time step are classified using a stage classification head to obtain the stage label of the robot's current stage.

[0035] Based on the stage labels, the robot's single-support phase and double-support phase are switched and controlled.

[0036] A second aspect of this application provides a robot whole-body motion control device, including:

[0037] The multimodal feature sequence acquisition module is used to acquire the robot's multimodal feature sequences.

[0038] The sub-target determination module is used to determine the robot's sub-targets based on the multimodal feature sequence. The sub-targets are used to characterize the robot's expected action information at the current stage.

[0039] The collaborative control module is used to collaboratively control the robot's upper and lower limb movements based on sub-goals.

[0040] A third aspect of this application provides a robot including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot whole-body motion control method provided in the first aspect of this application.

[0041] A fourth aspect of this application provides a computer program product that, when run on a robot, causes the robot to perform the robot whole-body motion control method provided in the first aspect of this application.

[0042] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot whole-body motion control method provided in the first aspect of this application.

[0043] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0044] Figure 1 This is a flowchart of a robot whole-body motion control method provided in an embodiment of this application;

[0045] Figure 2 This is a schematic diagram of an operation process for controlling the upper limb movements of a robot based on sub-targets, provided in an embodiment of this application.

[0046] Figure 3 This is a schematic diagram of an operation process for controlling the lower limb movements of a robot based on sub-targets, provided in an embodiment of this application.

[0047] Figure 4 This is a schematic diagram of another operation process for controlling the lower limb movements of a robot based on sub-targets, provided in an embodiment of this application.

[0048] Figure 5 This is a schematic diagram of an operation process for correcting the footprint sequence and centroid trajectory based on external force / torque, provided in an embodiment of this application.

[0049] Figure 6 This is a schematic diagram of an operation process for adjusting the proportion of single-support phase and double-support phase of a robot according to a stage label, provided in an embodiment of this application.

[0050] Figure 7 This is a schematic diagram of the structure of a robot whole-body motion control device provided in an embodiment of this application;

[0051] Figure 8 This is a schematic diagram of a robot provided in an embodiment of this application. Detailed Implementation

[0052] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0053] VLA models can directly generate executable motion control commands based on the robot's multimodal feature inputs to control the robot to complete corresponding embodied intelligence tasks. However, if the task to be executed is an embodied intelligence task with a multi-stage, long-sequential motion chain, such as an object handling task involving approaching, grasping, lifting, transferring, placing, and removing objects, the robot will experience large load changes and environmental disturbances at different stages. Ultimately, this will make it difficult for the robot to maintain steady-state coordination of its upper and lower limb movements, and it is prone to instability risks.

[0054] To address the aforementioned technical problems, this application provides a robot whole-body motion control method, a robot, and a computer program product. This method enables the robot to maintain steady-state coordination of its upper and lower limb movements during long motion chain tasks, reducing the risk of instability. For more specific technical implementation details of this application's embodiments, please refer to the various method embodiments described below.

[0055] Please see Figure 1 This application illustrates a robot motion control method according to an embodiment, comprising:

[0056] 101. Obtain the robot's multimodal feature sequence;

[0057] It should be understood that the executing entity of this method embodiment can be any type and model of robot, such as humanoid robot or non-humanoid robot, which can complete embodied intelligent tasks with long action links, including item handling, item sorting, item selection, shelf loading and unloading, tool retrieval and placement, door opening and closing, and drawer pulling.

[0058] When a robot performs an embodied intelligent task, its multimodal feature sequences are acquired in real time. The multimodal feature data acquired by the robot may include, but is not limited to: information such as joint position and joint velocity collected by joint encoders; information such as robot posture, angular velocity, and linear acceleration collected by the torso inertial measurement unit; information such as contact force and gripping force collected by force sensors / torque sensors on the robot's feet or grippers; and image sequences of the task scene environment and the object to be manipulated collected by monocular, binocular, or RGB-D cameras, etc. When the robot performs an embodied intelligent task, the acquired multimodal feature data is mainly collected in real time by the robot itself and its configured environmental sensors during operation. However, in the robot's motion control model training or playback mode, pre-recorded historical multimodal sensor data can be used. The specific methods for acquiring the multimodal feature sequences are described below.

[0059] In one implementation of this application, obtaining the multimodal feature sequence of the robot includes:

[0060] (1) Obtain the robot's visual feature sequence, state feature sequence, and semantic vector obtained by parsing task action instructions;

[0061] (2) Concatenate the visual feature sequence, state feature sequence and semantic vector to obtain the multimodal feature sequence.

[0062] The robot is equipped with a task language encoder, which can parse the user's input natural language task action instructions into a fixed-dimensional structured intent vector and constraint template. For example, it is used to represent semantic information and soft constraints such as "which box to move, which storage location to move, whether to keep it horizontal, whether to handle it gently, place it on the left, do not tip it over, and the target layer and orientation", thereby obtaining a semantic vector z. lang The robot is also equipped with a perception and state encoder, which can extract visual temporal features and low-dimensional state temporal features (such as joint data, IMU data, tactile or torque data, foot contact markers, etc.). Specifically, each frame of image acquired by the visual sensor can be encoded into a corresponding feature vector through a convolutional neural network or a visual Transformer network. Concatenating these feature vectors in chronological order yields the visual feature sequence z. vis The joint angles, joint velocities, IMU data, and force sensor data detected at each time step by the state sensors can all be encoded into corresponding state feature sequences z using small networks such as multilayer perceptrons. state After obtaining the robot's visual feature sequence, state feature sequence, and semantic vector within the observation time window, the corresponding visual features, state features, and semantic vector are concatenated into a high-dimensional vector at each time step, thus obtaining the robot's multimodal feature sequence (z).lang , z vis , z state As can be seen, multimodal feature sequences integrate various features such as vision, state, and task semantics, and can be used to achieve high-precision full-body motion control of robots.

[0063] 102. Based on the multimodal feature sequence, determine the robot's sub-objectives. The sub-objectives are used to characterize the robot's expected action information at the current stage.

[0064] After acquiring the robot's multimodal feature sequence, the robot's sub-targets can be determined by analyzing the multimodal feature sequence. The sub-targets are used to characterize the robot's expected action information at the current stage, that is, the stable action that the robot expects to complete when performing the task at the current stage, such as the expected target end pose, the gripper's grasping pose, the object placement posture and orientation, contact state, placement position, object contact time, and object release time, etc.

[0065] In one implementation of this application, determining the robot's sub-targets based on a multimodal feature sequence includes:

[0066] (1) Input the multimodal feature sequence into the trained temporal fusion neural network to perform feature fusion processing for the time dimension and obtain the latent vectors of each time step;

[0067] (2) The latent vectors at each time step are regressed using the sub-target regression head to obtain the sub-target.

[0068] Multimodal feature sequences can be input into a trained temporal fusion neural network to perform feature fusion processing along the time dimension, thereby obtaining the latent vectors at each time step. Specifically, a Transformer network or a bidirectional LSTM network is pre-trained as the aforementioned temporal fusion neural network, which can be called the temporal backbone network. This temporal backbone network achieves cross-modal and cross-temporal information interaction and fusion through self-attention or recurrent structures. After the multimodal feature sequences are input into this temporal backbone network, the latent vector z_t at each time step is output by fusing the multimodal feature vectors at each time step.

[0069] Furthermore, the aforementioned temporal backbone network can have multiple output heads, connecting at least two types of output heads to each latent vector z_t: a stage classification head and a sub-target regression head. The stage classification head performs multi-class classification on the latent vector z_t, outputting the task stage to which the current time step belongs. Based on this, the robot's current stage label can be determined, such as labels for approach, grasping, lifting, transporting, placing, and withdrawing. The sub-target regression head performs regression analysis on the latent vector z_t, outputting the control sub-targets under the current stage, such as the end effector / manipulator target pose, object target posture, desired center of gravity position, contact mode, and contact / release time. During the training of the aforementioned temporal backbone network, manually or automatically labeled "stage labels" and "sub-target labels" are used as supervision signals, enabling the temporal backbone network to automatically complete stage recognition and sub-target prediction given multimodal feature sequence input. Using the output stage labels and sub-targets, the robot can achieve precise and flexible coordinated control of upper and lower limb movements; specific control methods are described below.

[0070] 103. Based on sub-goals, coordinate the upper and lower limb movements of the robot.

[0071] After determining the sub-objective for robot motion control in the current stage based on the multimodal feature sequence, the robot's upper and lower limb movements are coordinated and controlled based on this sub-objective. Upper limb movements mainly include the movements of upper limb joints and mechanical grippers, while lower limb movements mainly include the movements of lower limb joints and robot feet. Since both upper and lower limb movements are controlled based on the same sub-objective for the current stage, it is possible to simultaneously achieve the desired stable movements for the corresponding stage, thus maintaining steady-state coordination of the robot's upper and lower limb movements and reducing the risk of instability. For example, in the object lifting stage, the sub-objective for robot motion control could be slow and stable lifting. In this case, the robot's upper limbs are controlled to slowly lift the object, while the lower limbs are coordinated and controlled to stand still, thereby achieving steady-state coordination and significantly reducing the risk of robot instability. The following describes in detail the specific implementation method of controlling the robot's upper and lower limb movements based on sub-objective coordination.

[0072] In one implementation of this application, the coordinated control of the robot's upper and lower limb movements based on sub-objectives includes:

[0073] (1) Determine the desired pose of the robot's upper limb end based on the sub-objective;

[0074] (2) The desired pose is converted into the desired state of the robot's upper limb joints by inverse kinematics;

[0075] (3) Generate a reference motion trajectory for the upper limb joint based on the current and desired states of the upper limb joint;

[0076] (4) Control the upper limb joints according to the reference movement trajectory.

[0077] A robot can pre-train an upper limb manipulation head, which generates reference commands in the robot's joint space or task space based on sub-objectives and controls the robot's upper limb joints based on these reference commands. Specifically, the desired pose of the robot's upper limb end effector can be determined based on the sub-objectives, such as the grasping pose of a robotic gripper on an object. Then, through inverse kinematics (e.g., analytical IK or small network approximation IK), the desired pose is converted into the desired states of the robot's upper limb joints, such as desired angles and velocities. Next, based on the current and desired states of the upper limb joints, a reference motion trajectory is generated. This can be achieved by smoothing interpolation or trajectory planning between the current and desired states, such as performing polynomial or spline curve trajectory planning, to obtain a continuous reference motion trajectory for the upper limb joints, which can serve as the aforementioned reference commands. Finally, the robot's upper limb joints are controlled according to the obtained reference motion trajectory, meaning the actual motion trajectory of the upper limb joints follows the changes in the reference motion trajectory. This setup allows the robot's upper limb joint movements to be controlled to meet the requirements of the sub-objectives.

[0078] In one implementation of this application, the coordinated control of the robot's upper and lower limb movements based on sub-objectives includes:

[0079] (1) Determine the opening and closing degree and expected gripping force of the robot's mechanical gripper in the corresponding time period based on the contact time markers contained in the sub-targets;

[0080] (2) Control the mechanical gripper according to the degree of opening and closing and the desired clamping force.

[0081] The aforementioned upper limb manipulation strategy head can also generate a gripping plan for the robot's mechanical gripper, i.e., a planned gripping action, based on sub-goals. Specifically, sub-goals include contact time markers such as "when to start gripping," "when to start holding," and "when to start releasing." Based on these contact time markers, the strategy head can determine and output the opening and closing degree of the mechanical gripper and the desired gripping force within the corresponding time period. Then, the mechanical gripper can be controlled according to the determined opening and closing degree and desired gripping force. For example, the mechanical gripper can slowly close as it approaches an object, maintain a certain gripping force after firmly grasping the object, and slowly reduce the gripping force and release before placing the object, and so on. Additionally, if the sub-goal contains semantics such as "gentle placement," the mechanical gripper will reduce its speed and acceleration as it approaches the placement position, making the gripping and releasing action smoother. Through this setting, the robot's mechanical gripper actions can be controlled to meet the requirements of the sub-goals.

[0082] As an example, Figure 2 This is a schematic diagram of the operation process of controlling the upper limb movements of a robot based on sub-targets, provided in an embodiment of this application. Figure 2 The upper limb control objects shown include upper limb joints and a robotic gripper. For the upper limb joints, the desired pose of the upper limb end is first determined based on the sub-objective. Then, through inverse kinematics, the desired pose is converted into the desired state of the upper limb joint. Next, smooth interpolation or trajectory planning is performed between the current state and the desired state of the upper limb joint to generate a reference motion trajectory. Finally, the actual motion trajectory of the upper limb joint is controlled to follow the reference motion trajectory. For the robotic gripper, the opening and closing degree and desired gripping force of the robotic gripper within the corresponding time period are first determined based on the contact time markers contained in the sub-objective. Then, by combining various semantics in the sub-objective with the determined opening and closing degree and desired gripping force, the robotic gripper is controlled to complete corresponding actions such as grasping and releasing objects.

[0083] As can be seen from the above, when controlling the robot's upper limb movements based on sub-goals, the sub-goals are specifically implemented as executable upper limb end / joint movement trajectories, as well as the action plans for opening and closing of the mechanical gripper and clamping force. This successfully achieves the stable mapping of the task intent of natural language into real end / joint commands, as well as the grasping, opening and closing and contact timing plans, thereby avoiding problems such as semantic correctness but distortion of the real physical situation in the robot's motion control process.

[0084] The above describes the relevant content of controlling the upper limb movements of a robot based on sub-targets. Next, we will describe the relevant content of controlling the lower limb movements of a robot based on sub-targets.

[0085] In one implementation of this application, the coordinated control of the robot's upper and lower limb movements based on sub-objectives includes:

[0086] (1) Generate the robot's expected footprint sequence and gait phase based on the sub-target and obstacle information in the robot's environment;

[0087] (2) Generate the desired zero-moment point trajectory that is always within the support polygon based on the desired footprint sequence and gait phase;

[0088] (3) Convert the desired zero-moment point trajectory into the desired centroid trajectory;

[0089] (4) Control the robot’s lower limb movements according to the desired centroid trajectory so that the robot’s actual centroid trajectory follows the desired centroid trajectory.

[0090] The robot can pre-train a lower limb locomotion stabilizer, which focuses on intrinsic momentum / center of mass regulation, footprint and gait phase generation, and contact timing management. It employs methods such as proportional-derivative control (PD), model predictive control (MPC), and hierarchical quadratic programming (HQP) to achieve stable support and adaptive footprinting for the robot's lower limbs during standing, walking, and transport. Specifically, based on the sub-objective and obstacle information in the robot's environment, a path planning algorithm can generate a movement trajectory that satisfies the requirements of the sub-objective and successfully avoids obstacles, along with the support phase at each time step, thereby obtaining the robot's desired footprint sequence and gait phase. The momentum / center of mass adjustment process then begins: Based on the desired footprint sequence and gait phase, an ideal desired zero-moment point (ZMP) trajectory that remains within the support polygon is generated. Then, using a linear inverted pendulum model or a simplified momentum model, the desired ZMP trajectory is converted into a desired center of mass (CoM) trajectory. This CoM trajectory contains information such as the position and velocity of the desired center of mass. Finally, the robot's lower limb movements are controlled according to the desired CoM trajectory, ensuring that the robot's actual CoM trajectory follows the desired CoM trajectory. During the execution of the robot's lower limb movements, the actual CoM trajectory is continuously monitored. If the actual CoM trajectory deviates from the desired CoM trajectory, or if the predicted capture point approaches the boundary of the support polygon, it indicates a risk of robot instability. In this case, the robot's next footprint position or gait phase duration is fine-tuned, for example, by adjusting the footprint position to take a step outward or increasing the duration of the double support phase, thereby reducing the risk of robot instability. When the footprint position and / or gait phase are adjusted, the desired zero-moment point trajectory will be adjusted, and the corresponding desired center-of-mass trajectory will also be adjusted. The stabilizer uses the adjusted desired center-of-mass trajectory as a reference target and transmits it to the rear control module of the robot's lower limbs. Through torque or joint acceleration control, the robot's actual center-of-mass trajectory continuously tracks the adjusted desired center-of-mass trajectory. It can be seen that the essence of the momentum / center-of-mass adjustment process is to use a simplified model to predict the robot's potential instability risk, and then compensate in advance by adjusting the desired center-of-mass trajectory and footprint plan. This can effectively improve the stability of the robot's lower limb movements and reduce the risk of instability.

[0091] As an example, Figure 3 This is a schematic diagram illustrating the operation flow of a robot's lower limb movements based on sub-target control, as provided in an embodiment of this application. Figure 3In the process of momentum / center of mass adjustment, the robot first combines the sub-target and obstacle information of the robot's environment to generate discrete expected footprint sequences and gait phases based on path planning algorithms. Then, based on the expected footprint sequence and gait phase, an ideal expected zero-moment point trajectory that always stays within the supporting polygon is generated. The expected zero-moment point trajectory is converted into an expected center of mass trajectory using a linear inverted pendulum model or a simplified momentum model. If the actual center of mass trajectory deviates from the expected center of mass trajectory or the predicted capture point is close to the boundary of the supporting polygon, the position of the next footprint or the duration of the gait phase is adjusted and the expected center of mass trajectory is updated synchronously. Then, the adjusted expected center of mass trajectory is used as a reference target to control the actual center of mass trajectory to continuously track the adjusted expected center of mass trajectory.

[0092] In one implementation of this application, controlling the robot's lower limb movements according to a desired centroid trajectory includes:

[0093] (1) Construct an objective function to minimize the error between the actual footprint sequence and the expected footprint sequence of the robot, the error between the actual centroid trajectory and the expected centroid trajectory, and the lower limb joint torque of the robot;

[0094] (2) Construct constraints for the robot’s foot parameters, friction cone parameters, and lower limb joint parameters;

[0095] (3) Under the condition of satisfying the constraints and minimizing the objective function, solve for the control parameters of the robot's lower limb joints;

[0096] (4) Control the lower limb joints according to the control parameters.

[0097] The lower limb locomotion stabilizer can also employ methods such as proportional-derivative (PD), model predictive control (MPC), or hierarchical quadratic programming (HQP) to perform real-time tracking and force distribution control of the robot's lower limb joints / endpoints, thereby ensuring stability within the supporting polygon. Specifically, given information such as the robot's desired footprint sequence, actual footprint sequence, desired centroid trajectory, actual centroid trajectory, and gait phase, an optimization problem based on whole-body optimization or MPC can be created. This problem constructs an objective function to minimize the error between the robot's actual footprint sequence and desired footprint sequence, the error between the actual centroid trajectory and desired centroid trajectory, and the robot's lower limb joint torques. Furthermore, constraints are constructed to constrain the robot's foot parameters, friction cone parameters, and lower limb joint parameters. For example, the constructed constraints may include, but are not limited to: zero velocity at the supporting foot and zero force at the non-supporting foot, ground reaction force not exceeding the friction cone force, specific joint position and torque range constraints, etc. Next, under the condition of satisfying the constraints and minimizing the objective function, the control parameters of the robot's lower limb joints are solved. Finally, the robot's lower limb joints are controlled according to the solved control parameters. For example, the torque or acceleration parameters of each lower limb joint of the robot can be solved under the condition of satisfying the constraints and minimizing the objective function. The lower limb joints are then controlled according to these torque and acceleration parameters, so that the robot's actual footprint sequence follows the desired footprint sequence, the robot's actual center of mass trajectory follows the desired center of mass trajectory, and the contact and stability requirements of the robot's motion are met.

[0098] As an example, Figure 4 This is a schematic diagram of another operation process for controlling the lower limb movements of a robot based on sub-targets, provided in an embodiment of this application. Figure 4 This describes the process of real-time tracking and force distribution control of a robot's lower limb joints: Given the robot's desired footprint sequence, actual footprint sequence, desired centroid trajectory, actual centroid trajectory, and gait phase, an optimization problem is created. The objective function of this optimization problem minimizes the footprint sequence error, centroid trajectory error, and joint torque magnitude. Furthermore, constraints including contact constraints, friction cone constraints, joint position, and torque range are constructed. While satisfying the constraints and minimizing the objective function, control parameters such as torque or acceleration for each lower limb joint are solved, and the corresponding lower limb joints are controlled according to the obtained control parameters.

[0099] In one implementation of this application, the method further includes:

[0100] (1) Detect the external forces and torques acting on the robot's upper limb end;

[0101] (2) Based on the external force and external torque, correct the expected footprint sequence and the expected centroid trajectory.

[0102] During the coordinated control of the robot's upper and lower limbs, the external forces and torques acting on the robot's upper limb endcaps can be detected in real time and input as feedforward terms to the aforementioned lower limb locomotion stabilizer. This can be used to synchronously correct the desired footprint sequence and desired center of mass trajectory, thereby reducing the degree of center of mass drift during lifting, turning, and placement phases. Specifically, if force sensors / torque sensors are installed on the robot's upper limb endcaps, the external forces and torques acting on the robot's upper limb endcaps can be directly measured through the sensors. If force sensors are not installed on the robot's upper limb endcaps, the external forces acting on the endcaps can be estimated by combining joint drive current or torque sensors with a dynamic model. Then, based on the geometric relationship between the upper limb endcaps and the robot's center of mass, the external forces and torques acting on the endcaps can be converted into equivalent external forces and torques acting on the robot as a whole. When correcting the desired footprint sequence and desired centroid trajectory based on external forces and torques, the detected external forces can be regarded as perturbation terms in the centroid / momentum model. If the external force causes a change in the centroid position or a change in the overall force direction, corresponding compensation can be added to the desired centroid trajectory, such as slightly moving the centroid reference position in the opposite direction. If the current zero torque point or capture point is predicted to be close to the boundary of the support polygon, the next footprint position can be adjusted to step outward or the duration of the double support phase can be extended, and so on. By adopting this mechanically consistent feedforward trajectory correction mechanism, the additional load generated by the robot's upper limbs will be fed back to the lower limb stabilizer in advance, enabling the desired centroid trajectory and footprint sequence to actively make adaptive corrections based on the external forces acting on the upper limbs, thereby further reducing the risk of instability of the robot when performing critical actions such as lifting, turning, and placing.

[0103] As an example, Figure 5 This is a schematic diagram of an operation process for correcting footprint sequences and centroid trajectories based on external force / torque, provided in an embodiment of this application. Figure 5 First, by combining force sensors, torque sensors, and a dynamic model, the external forces and torques acting on the robot's upper limb end are estimated. The estimated external forces and torques are then treated as perturbation terms in the center of mass / momentum model. If the perturbation terms cause a change in the position of the center of mass or a change in the overall force direction, corresponding compensation can be added to the expected center of mass trajectory to correct the trajectory. If the perturbation terms cause the predicted zero torque point or capture point to be close to the boundary of the support polygon, the position of the next footprint or the duration of the double support phase is adjusted to correct the footprint sequence.

[0104] In one implementation of this application, the method further includes:

[0105] Based on the stage labels, the robot's single-support phase and double-support phase are switched and controlled.

[0106] Referring to the previous description, the stage classification head of the temporal backbone network can output the robot's current stage label, which indicates which of the following stages the current time step belongs to: approach, grasp, lift, transfer, placement, or withdrawal. During training, this temporal backbone network labels the teaching or simulation data into stages, and after training, it can automatically output the stage label for the current stage based on current observations and historical information. Downstream modules such as the robot's upper and lower limb control strategies, the single support phase (SSP) / dual support phase (DSP) ratio selection, and safety threshold settings can adopt different configurations according to different stage labels. For example, stability priority can be increased in the grasp / placement stage, while walking efficiency priority can be increased in the transfer stage, and so on. It can be seen that the stage label is an important signal for coordinating when the robot's movements are more stable and when they are faster. Specifically, based on the robot's current stage label, the robot's single support phase (SSP) and dual support phase (DSP) can be switched accordingly. For example, when the stage label is "Approach," the footprint planning focuses on avoiding obstacles and approaching the target object. The robot's upper limbs mainly perform rough alignment without large grasping movements. In this case, the ratio of the single support phase (SSP) and the dual support phase (DSP) can be kept relatively balanced. When the stage label is "Grasp," gait control forces the robot into the dual support phase (DSP) or maintains it for most of the time to improve stability, keeping the robot's desired center of mass trajectory near the center of the support polygon. The upper limbs perform fine grasping movements, and the mechanical gripper gradually closes. When the stage label is "Transfer," the ratio of the single support phase (SSP) and the dual support phase (DSP) is adaptively adjusted according to the load weight and ground friction. For example, under light load and high friction, the proportion of the single support phase (SSP) can be increased to improve speed, while under heavy load or low friction, the proportion of the dual support phase (DSP) can be increased to reduce speed and the risk of instability. When the stage label is "Placement," the proportion of the dual support phase (DSP) can be further increased, or even maintained throughout the entire process. The robot's upper limbs slowly lower the height of the object, reducing the end effector velocity and acceleration, and the mechanical gripper smoothly releases, completing the action of steadily and gently placing the object. It can be seen that by combining the linkage control strategy with stage labels, the robot can automatically adjust its steady-state and motion strategies according to the characteristics of different task stages, thereby achieving the technical effect of being more stable when it should be stable and faster when it should be fast.

[0107] As an example, Figure 6 This is a schematic diagram illustrating an operation process for adjusting the proportion of single-support and double-support phases of a robot based on stage labels, as provided in an embodiment of this application. Figure 6In the process, when the stage label is "Approach," the proportion of control for the single support phase (SSP) and the dual support phase (DSP) is basically balanced; when the stage label is "Grasp," the proportion of control for the dual support phase (DSP) is much higher than that for the single support phase (SSP); when the stage label is "Transfer," the proportion of control for the single support phase (SSP) and the dual support phase (DSP) is adaptively adjusted according to the load weight and ground friction conditions. Specifically, the proportion of single support phase (SSP) can be increased under light load and high friction, while the proportion of dual support phase (DSP) can be increased under heavy load or low friction; when the stage label is "Placement," the proportion of control for the dual support phase (DSP) is much higher than that for the single support phase (SSP). In addition to the upper and lower limb coordinated control strategy described above, the technical solution of this application embodiment can also superimpose a residual safety control layer, superimpose impedance / force control residuals on the robot's whole-body control commands, and ensure the safety and compliance of the robot's end-effector speed, grasping normal force, human-machine distance, and contact switching through constraint projection and other methods. Constraint projection refers to making the actual control quantity as close as possible to the reference control quantity without violating safety constraints. For example, impedance / force control residuals can be superimposed on the full-body motion control commands to be issued to the robot and constraint projection can be applied to ensure that the robot's end effector speed does not exceed the speed limit v. max (e.g., 0.4 m / s), the grasping normal force does not exceed the upper limit of force f. max (It can be adaptively adjusted according to the weight of the object that the robot needs to operate), and the human-machine distance is not less than the lower limit d. min (e.g., 0.25m), and maintaining smooth contact switching, the specific operating method is described below.

[0108] The residual safety control layer is equivalent to adding another layer of safety protection on top of the existing reference control variables for the upper and lower limbs. First, impedance / force control residuals ensure the robot joints respond compliantly to deviations and external forces. Then, a small-scale quadratic programming (QP) or projection problem is constructed to adjust the control variables to meet safety constraints, including upper limits on velocity, grasping normal force, minimum human-robot distance, and smooth contact transitions. Specifically, when solving the quadratic programming problem, the reference control variables can be projected onto a set of movable actions C. The control variables within set C must simultaneously satisfy the following conditions:

[0109] (1) The speed of each critical point at the end of the robot does not exceed its respective speed limit v. max , can be represented as in Indicates the velocity of the terminal key point i;

[0110] (2) The grasping normal force at each contact point of the robot does not exceed its respective force limit f. max , can be represented as ||f j ||2≤fmax，j , where f j This represents the gripping normal force at contact point j;

[0111] (3) The distance between the human and the machine shall not be less than the lower limit of distance d. min , can be represented as d human (q)≥d min , where d human (q) represents the real-time changing distance between the robot and the human body;

[0112] (4) Maintain smooth contact switching.

[0113] If a hard constraint is violated (e.g., a joint torque is significantly exceeded), the corresponding control quantity is forcibly corrected within a single step, the event is recorded, and if necessary, the upper-level strategy execution mode is notified to switch or emergency actions such as deceleration and shutdown are performed. In summary, the technical solution of this application embodiment ensures that the robot can walk stably through trajectory tracking and force distribution, and then ensures that the safety boundary of motion control is not breached through the residual safety control layer.

[0114] Taking joint space impedance control as an example, the impedance / force control residual can be expressed by the following formula:

[0115]

[0116] Where, τ t This indicates the actual torque applied to the joint. Indicates the reference torque. K represents the impedance residual. p Indicates the stiffness coefficient. q represents the desired joint position. t K represents the actual joint position. d Indicates the damping coefficient. Indicates the desired joint velocity. K represents the actual joint velocity. p and K d It can distinguish between upper and lower limbs for separate tuning. In principle, the above formula is equivalent to connecting a virtual spring and a virtual damper in series on the reference torque. When the joint deviates from the desired trajectory, the virtual spring and virtual damper can generate a corresponding restoring torque, while also absorbing energy from rapid changes and external impacts, making the robot's whole-body motion control smoother and more stable.

[0117] In practice, assuming the reference control variable for the robot's full-body motion to be issued is u_ref, and the safety control variable is u_safe, then we first construct an objective function that minimizes the deviation between the reference control variable and the safety control variable: ||u_safe–u_ref|| 2Based on this, constraints are constructed to limit the robot's joint parameters, end effector velocity, grasping normal force, and distance from the human body. For example, the velocity and torque of each joint can be set to be within allowable ranges, and the end effector linear velocity can be set to not exceed a set upper limit v. max The grasping normal force does not exceed the set upper limit f max And the predicted distance between the human body and the human body is not less than the lower limit of the safe distance d. min The objective function described above can be viewed as a standard quadratic programming (QP) problem. Therefore, by solving the quadratic programming problem, the optimal safety control quantity u_safe can be determined while satisfying the constraints and minimizing the objective function. The actual control quantity u_safe issued to the robot is the optimal safety control quantity u_safe, not the initial reference control quantity u_ref. With this setting, when the reference control quantity u_ref already satisfies the constraints, u_safe and u_ref are essentially consistent. Only when certain components violate the safety boundaries will u_ref be slightly adjusted in necessary dimensions. This ensures that all key physical quantities of the robot meet safety and compliance requirements while maintaining the original output of the upper and lower limb coordinated control strategy as much as possible.

[0118] After introducing a residual safety control layer, the priority of the robot's whole-body motion control strategy can be set in the following order from high to low: safety constraints, support stability, contact force feasibility, upper limb end-effector task, joint regularization, etc. By employing methods such as model predictive control or hierarchical quadratic programming control to allocate forces and poses within the feasible region, a balance between task completion and motion stability can be achieved. The robot can also incorporate certain degradation and fault-tolerance mechanisms: when the robot experiences insufficient computing power or observation data, priority can be given to ensuring gait phase and support stability, reducing visual resolution and upper limb trajectory bandwidth, and avoiding frequent contact switching.

[0119] Furthermore, the technical solution of this application embodiment can also use smoothing functions such as Sigmoid or spline curves to transition the contact markers of the robot's mechanical claw or foot, thereby limiting force and velocity transitions and reducing the impact of robot movements. Taking the mechanical claw grasping an object as an example, a contact weight w can be defined, with a value range of 0 to 1. When w = 0, it indicates no contact or almost no force, and when w = 1, it indicates full contact and bearing the main contact force. When contact is established, the contact weight w is not instantly changed from 0 to 1, but rather the value of w is slowly increased over a certain period of time (e.g., several hundred milliseconds) using smoothing functions such as Sigmoid or spline curves until w changes from 0 to 1. Similarly, when contact is released, the contact weight w is also smoothly controlled to change from 1 to 0 over a certain period of time. When calculating force distribution, the motion controller multiplies the contact weight w by the corresponding contact force, thereby causing the contact force to increase or decrease gradually, rather than changing instantaneously. Similarly, the robot's feet make contact with the ground. By gradually changing the contact weight w, the robot's feet can be controlled to land and lift off the ground slowly, effectively reducing impact and vibration and improving stability. Additionally, information such as stage / contact switching, out-of-bounds projection events, stability, and task metrics can be recorded, which can be used for subsequent failure attribution and engineering iteration in the robot's whole-body motion control.

[0120] In terms of training and data organization, simulation and live teaching data can be used to monitor the robot's upper limb commands, grasping and opening / closing, stage boundaries, and lower limb phase labeling data. During training, the aforementioned lower limb locomotion stabilizer can first be stabilized offline under scenarios of standing, squatting, and micro-load variations, then perturbed and randomized under random upper limb trajectory driving, and finally coupled with the walking phase to complete the coupling of walking and manipulation. Furthermore, various reinforcement learning algorithms can be used to fine-tune composite indicators such as the robot's task success rate, collision rate, and posture stability.

[0121] If the robot experiences missing temporal data during observation, processing strategies such as discarding data by timestamp, delayed fusion, cross-modal timestamp alignment, or masking the missing sensor data can be employed. Specifically, missing observation data refers to certain sensor data not being updated or exhibiting severely abnormal data quality across multiple consecutive control cycles. In such cases, the following strategies can be used: For camera data, if a new image frame is missing in a control cycle, the features of the previous frame can be reused; if the timestamp of an image frame is significantly delayed, that frame will not participate in the fusion of the current cycle to avoid introducing outdated information. For state sensor data, if the data update frequency is higher than the control frequency, multiple state records can be averaged or the most recent timestamp can be selected within a control cycle. If some sensor data is missing, a mask of 0 can be added to these sensor data, thus preventing them from participating in the fusion of the current cycle and avoiding impacting the accuracy of the observation data. Through these processing strategies, even with network jitter and sensor latency, the acquired multimodal data can be ensured to be time-aligned and of controllable quality as much as possible. Furthermore, the upper and lower limb functional models described above, such as the upper limb manipulation strategy head and the lower limb locomotion stabilizer, can adopt a parameter freezing and thawing training method, and introduce teacher and student models from knowledge distillation technology to achieve phased training, thereby improving the stability of model convergence.

[0122] The technical solution of this application first obtains the multimodal feature sequence of the robot, then determines the robot's sub-targets based on the multimodal feature sequence. These sub-targets characterize the robot's expected action information at the current stage. Finally, based on these sub-targets, the robot's upper limb and lower limb movements are coordinated and controlled. With this setup, the robot's upper and lower limb movements can be controlled at each stage of a long motion chain task to simultaneously meet the stable expected actions of the corresponding stage. This maintains steady-state coordination of the robot's upper and lower limb movements at each stage, reducing the risk of instability.

[0123] In summary, the embodiments of this application, by employing a system design that incorporates stage perception, upper and lower limb layering, safety constraint projection, and steady-state coordination, enable the robot's upper limb control strategy and lower limb stabilizer to form a closed loop of consistent information and mechanics, thereby stably completing various long-motion chain tasks from end to end while meeting safety and compliance requirements.

[0124] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0125] The above mainly describes a method for controlling the whole body motion of a robot. The following will describe a device for controlling the whole body motion of a robot.

[0126] Please see Figure 7 One embodiment of a robot whole-body motion control device applied to the above-described motion control method in this application includes:

[0127] The multimodal feature sequence acquisition module 701 is used to acquire the multimodal feature sequence of the robot;

[0128] The sub-target determination module 702 is used to determine the robot's sub-targets based on the multimodal feature sequence. The sub-targets are used to characterize the robot's expected action information at the current stage.

[0129] The collaborative control module 703 is used to collaboratively control the upper and lower limb movements of the robot based on sub-goals.

[0130] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a robot whole-body motion control method as described in any of the above embodiments.

[0131] This application also provides a computer program product that, when run on a robot, causes the robot to perform a robot whole-body motion control method as described in any of the above embodiments.

[0132] Figure 8 This is a schematic diagram of a robot provided in one embodiment of this application. Figure 8 As shown, the robot 8 in this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, it implements the steps in the embodiments of the various robot whole-body motion control methods described above, for example... Figure 1 Steps 101 to 103 are shown. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 7 The functions of modules 701 to 703 are shown.

[0133] The computer program 82 can be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 82 in the robot 8.

[0134] The processor 80 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0135] The memory 81 can be an internal storage unit of the robot 8, such as a hard drive or memory. The memory 81 can also be an external storage device of the robot 8, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 81 can include both internal and external storage units of the robot 8. The memory 81 is used to store the computer program and other programs and data required by the robot 8. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0144] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for controlling the whole-body motion of a robot, characterized in that, include: Obtain the robot's multimodal feature sequence; Based on the multimodal feature sequence, the sub-targets of the robot are determined, and the sub-targets are used to characterize the robot's expected action information at the current stage; Based on the aforementioned sub-objectives, the robot's upper and lower limb movements are controlled collaboratively.

2. The method as described in claim 1, characterized in that, The coordinated control of the robot's upper and lower limb movements based on the sub-target includes: Based on the sub-objective, determine the desired pose of the robot's upper limb end; The desired pose is converted into the desired state of the robot's upper limb joints by inverse kinematics. Based on the current state and the desired state of the upper limb joint, a reference motion trajectory for the upper limb joint is generated; The upper limb joints are controlled according to the reference motion trajectory.

3. The method as described in claim 1, characterized in that, The coordinated control of the robot's upper and lower limb movements based on the sub-target includes: Based on the contact time markers contained in the sub-targets, determine the opening and closing degree and expected gripping force of the robot's mechanical gripper within the corresponding time period; The mechanical gripper is controlled according to the opening and closing degree and the desired clamping force.

4. The method as described in claim 1, characterized in that, The coordinated control of the robot's upper and lower limb movements based on the sub-target includes: Based on the sub-target and the obstacle information in the robot's environment, the expected footprint sequence and gait phase of the robot are generated; Based on the desired footprint sequence and the gait phase, generate a desired zero-moment point trajectory that always lies within the supporting polygon; Convert the desired zero-moment point trajectory into the desired centroid trajectory; The robot's lower limb movements are controlled according to the desired centroid trajectory, so that the robot's actual centroid trajectory follows the desired centroid trajectory.

5. The method as described in claim 4, characterized in that, The step of controlling the robot's lower limb movements according to the desired centroid trajectory includes: Construct an objective function to minimize the error between the robot's actual footprint sequence and the desired footprint sequence, the error between the actual centroid trajectory and the desired centroid trajectory, and the lower limb joint torque of the robot; Construct constraints to constrain the robot's foot parameters, friction cone parameters, and lower limb joint parameters; Solve for the control parameters of the robot's lower limb joints while satisfying the constraints and minimizing the objective function; The lower limb joints are controlled according to the control parameters.

6. The method as described in claim 4, characterized in that, The method further includes: Detect the external forces and torques acting on the robot's upper limb extremities; Based on the external force and the external torque, the desired footprint sequence and the desired centroid trajectory are corrected.

7. The method according to any one of claims 1 to 6, characterized in that, Determining the robot's sub-targets based on the multimodal feature sequence includes: The multimodal feature sequence is input into a trained temporal fusion neural network to perform feature fusion processing for the time dimension, thereby obtaining the latent vectors for each time step. The sub-target is obtained by performing regression processing on the latent vectors of each time step using a sub-target regression head.

8. The method as described in claim 7, characterized in that, The method further includes: The hidden vectors of each time step are classified using a stage classification head to obtain the stage label of the robot's current stage. Based on the stage labels, the robot's single-support phase and double-support phase are switched and controlled.

9. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the robot whole-body motion control method as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, When the computer program product is run on the robot, it causes the robot to perform the robot whole-body motion control method as described in any one of claims 1 to 8.

Images