A Humanoid Robot Motion Control Method Based on Footprint Planning

Abstract

This invention belongs to the field of robot control technology, specifically relating to a humanoid robot motion control method based on footprint planning. First, a policy network is constructed, including an input layer, two cascaded recurrent hidden layers, and a linear output layer. Then, a reward function is designed. Next, the constructed policy network is trained using the Mujoco simulation platform based on the reward function, initializing the robot's posture and terrain. Different terrains have different preset footstep sequences. The policy network is updated using the PPO algorithm. Finally, the trained policy network is used for humanoid robot motion control. This invention achieves stable walking across terrain under known target footstep sequences through a policy network and a sliding window observation mechanism, realizing closed-loop control from environment understanding to action generation. The designed reward function effectively guides the robot to form stable and natural gait behavior, and the trained policy network can be directly transferred to an actual robot platform.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, specifically relating to a motion control method for humanoid robots based on footprint planning. Background Technology

[0002] With the continuous development of mobile robot control technology, humanoid robots possess human-like characteristics, enabling them to overcome various obstacles and maintain excellent mobility and stability in complex environments. However, controlling the stability of humanoid robots remains a challenge for bipedal robot technology. Currently, the following two methods are mainly used in the field of all-terrain walking control for bipedal robots: 1) Blind walking strategy: The robot does not perceive the external environment and relies solely on feedback information from inertial sensors, foot force sensors, etc., for local gait planning and control. This method has certain practicality in structured terrain, but when encountering abrupt or irregular terrain (such as rocks, slopes, steps), it cannot predict path risks, easily leading to robot instability and falls; 2) Imitation of expert data or reference trajectory tracking: This method relies on expert-designed motion libraries, trajectory optimizers, or human demonstration data to allow the robot to learn a set of trajectories that perform well on flat ground or specific terrain. Although the movements learned in this way are more natural, their generalization ability is limited when facing changing terrain, and they cannot dynamically adjust key parameters such as stride length, stride width, and foot placement.

[0003] Currently, there is a wealth of research on the leg motion control of humanoid robots. For example, Chinese Patent 202411252516.5 discloses an autonomous motion planning method for the legs of a humanoid robot, including: Step S1: Describing the task of the robot's legs when performing actions in a world coordinate system; Step S2: Defining the problem to be solved and deriving the dynamic equations of the humanoid leg two-link model; Step S3: Representing the robot's joint torque trajectory using a b-spline polynomial with a time scale factor and incorporating its physical limitations into the formula; Step S4: Based on the torque curve formula, transforming the torque curve planning problem into a constrained multi-objective function optimization problem by constructing multiple objective functions; Step S5: Solving the multi-objective optimization problem based on the standard dynamic model, simultaneously optimizing multiple objectives to obtain the optimal torque curve; Step S6: Based on the existing torque curve, designing a closed-loop control system to track the torque curve in joint space. Chinese Patent 202410688294.5 discloses a humanoid robot gait imitation learning method that combines periodic rewards. It constructs a reference action library that integrates contact information as a reference for imitation rewards and periodic contact rewards. A comprehensive reference action library is created by imitating the style and contact information of basic actions and their corresponding periodic contact information. By imitating the style of the reference actions and their contact information, and introducing periodic rewards, not only is the realism and style consistency of the robot's movements improved, but attention is also enhanced to the details of foot-ground interaction during action execution, thereby ensuring the stability of the humanoid robot's posture when walking in a human-like manner in a plane. Chinese Patent 202410617290.8 discloses a reinforcement learning-based method for generating multimodal adaptive gait of bipedal robots. First, a multimodal walking task is constructed, including simple planar tasks and complex non-planar tasks. Then, a gait generator is trained using a policy based on the concept of curriculum learning. Simultaneously, different reward functions are used to incentivize the training policy according to different multimodal gait tasks to obtain the optimal parameters of the gait generator. The gait generator is designed as a reinforcement learning framework jointly constructed using a dominant actor-critic framework (AC network) and a proximal policy optimization algorithm (PPO algorithm). Finally, the trained gait generator is used to generate a bipedal robot gait that satisfies the actual walking task.

[0004] However, the robots in the above-mentioned existing technologies all rely on reference trajectories, have many and complex training tasks, and cannot continuously switch between multiple walking modes according to real-time changes in road conditions. They also have poor gait control stability and insufficient generalization ability in complex terrain environments. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a humanoid robot bipedal motion control method based on footprint planning. This method directly integrates terrain perception and target foot trajectory planning results into a reinforcement learning strategy, which can effectively improve the robot's walking stability and landing accuracy in unstructured complex terrain, and has stronger environmental adaptability and task generalization ability.

[0006] To achieve the above objectives, the present invention provides a humanoid robot motion control method based on footprint planning, the specific steps of which include the following steps 1-4.

[0007] Step 1: Construct the policy network.

[0008] The policy network uses a long short-term memory neural network structure, which includes a four-layer structure: an input layer, two cascaded hidden layers, and a linear output layer.

[0009] Step 2: Design the reward function.

[0010] The reward function is shown in equation (1): (1).

[0011] In formula (1) To encourage the robot to have the correct ground contact force in the corresponding phase phase; To encourage the robot to land its feet accurately near the target point; To encourage the robot to maintain a reasonable speed during the swinging and supporting phases; To encourage the robot to maintain its vertical height at the standard orientation value; To punish the difference between the current action and the action in the previous moment; To punish energy consumption; To punish unreasonable collisions; , , , , , , All are weighting coefficients.

[0012] Step 3: Train the policy network.

[0013] Based on the above reward function, the constructed policy network is trained using the Mujoco simulation platform, and the policy network is updated using the PPO algorithm. First, the robot's posture and terrain are initialized, with different footstep sequences preset for different terrains. The robot's body state and footstep information for the next two time moments at the current moment, as well as periodic signals, are collected as inputs to the policy network. The policy network outputs the joint positions of 12 target actions. The reward function is used to evaluate the output results of the policy network, measuring the quality of the action or behavior results, and the policy network is updated using the PPO algorithm.

[0014] Step 4: Use the trained policy network to control the motion of the humanoid robot.

[0015] In the present invention formula (1) and This is to encourage robots to develop a symmetrical gait with a clear support-swing rhythm during training. To encourage sufficient ground contact force on the supporting feet during the support phase, ensuring the robot's stability during single-leg or double-leg support; The definition is as follows: ,in, and These represent the contact forces between the left and right feet and the ground in the current state. , This is the phase function used to adjust the contact force between the left and right feet during the support phase.

[0016] Formula (1) of this invention To encourage the swinging foot to maintain a reasonable forward swing speed during the swing phase and to avoid premature contact with the ground, The definition is as follows: ,in, and These represent the speeds of the left and right feet in the current state; , This is the phase function used to adjust the speed of the left and right feet during the swing phase.

[0017] In the present invention formula (1) The definition is as follows: ,in It is the height of the root in the world coordinate system in the current state. This is the reference height when standing statically.

[0018] In the present invention formula (1) The definition is as follows: ,in It is the distance between the next target foot and the nearest foot to it. It is the distance between the next target footstep and the root. It is a hyperparameter.

[0019] In the present invention formula (1) ,in, It represents the action of the i-th joint in its current state and the action at the previous moment.

[0020] In the present invention formula (1) ,in, These are the torque output at the current moment and the torque output at the previous moment for the i-th joint, respectively.

[0021] In the present invention formula (1) ,in, It refers to the impact force on the knee and hip joints.

[0022] The body state described in this invention includes joint angles and their velocities, trunk posture, and historical actions. The joints refer to the drive joints of the robot's legs, specifically including the hip joint, knee joint, and ankle joint. The trunk posture includes the three-dimensional position and three-dimensional posture of the robot's trunk. The historical actions are the joint target positions output by the policy network at the previous moment. The footstep information is the future step point data in the preset target footstep sequence. The periodic signal refers to phase time information.

[0023] During training, the present invention employs a window sliding mechanism to collect footstep information for the next two time points from the current moment.

[0024] This invention proposes a humanoid robot control framework that integrates gait planning and reinforcement learning, which can naturally achieve continuous switching between various walking modes (such as walking straight, turning, and walking backward). It designs a reward function system based on relative target foot points, which enables the robot to learn coordinated and task-oriented gait behavior without relying on reference trajectories.

[0025] Compared with existing technologies, this invention integrates footstep sequences with a low-level reinforcement learning-based policy control module. Under the condition of a known target footstep sequence, it achieves stable walking across terrain through a policy network and a sliding window observation mechanism, realizing closed-loop control from environmental understanding to action generation. Compared with traditional blind walking strategies, this invention has stronger planning capabilities and stability. Compared with expert trajectory imitation methods, this invention has higher flexibility and generalization ability on diverse terrains. By introducing factors such as foot landing hit rate, center of gravity balance, and gait rhythm into the reward function, it can effectively guide the robot to form stable and natural gait behavior. The entire policy training process can be completed in simulation, which has good engineering deployment value. The trained policy network can be directly transferred to an actual robot platform. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating the framework principle of the humanoid robot motion control method based on footprint planning involved in this invention.

[0027] Figure 2 This is a schematic diagram of the results of using the trained policy network of the present invention for motion control of a humanoid robot, where A represents jumping on rocks, B represents avoiding obstacles, C represents crossing gaps, and D represents walking on rough ground. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0029] Example 1

[0030] This embodiment relates to a motion control method for a humanoid robot based on footprint planning, and the specific steps are steps 1 to 4.

[0031] Step 1: Construct the policy network.

[0032] The policy network uses a Long Short-Term Memory (LSTM) neural network structure, which includes an input layer, two cascaded recurrent hidden layers, and a linear output layer of size 12. The robot's body state, the step information of the current moment and the next two moments, and the periodic signal are used as inputs. After passing through the policy network, the output is 12 target joint positions. The recurrent hidden layer contains 128 neurons, and 12 indicates that the network output is a 12-dimensional vector, corresponding to the target joint positions of the robot's 12 motion quantities.

[0033] Step 2: Design the reward function.

[0034] To guide the network strategy to form a physically plausible natural gait behavior and accurately use the step sequence as the robot's landing point, three main aspects are considered when designing the reward function: reasonable bipedal gait reward, reward for hitting the target step position, and energy consumption and other rewards that can maintain robot stability; the final reward function is shown in equation (1): (1), in equation (1) To encourage the robot to have the correct ground contact force in the corresponding phase phase; To encourage the robot to land its feet accurately near the target point; To encourage the robot to maintain a reasonable speed during the swinging and supporting phases; To encourage the robot to maintain its vertical height at the standard orientation value; To punish the difference between the current action and the action in the previous moment; To punish energy consumption; To punish unreasonable collisions; , , , , , , All are weighting coefficients.

[0035] In formula (1) and This is to encourage robots to develop a symmetrical gait with a clear support-swing rhythm during training. To encourage sufficient ground contact force on the supporting feet during the support phase, ensuring the robot's stability during single-leg or double-leg support; The definition is as follows: (2).

[0036] In formula (2) and These represent the contact forces between the left and right feet and the ground in the current state. , This is a phase function used to adjust the contact force between the left and right feet at different stages (support stage).

[0037] To encourage the swinging foot to maintain a reasonable forward swing speed during the swing phase and to avoid premature contact with the ground, The definition is as follows: (3).

[0038] In formula (3) and These represent the speeds of the left and right feet in the current state; , This is a phase function used to adjust the speed of the left and right feet during different phases (swing phase).

[0039] To prevent the robot's overall posture from sinking or deviating from the desired height, a reward for maintaining the robot's vertical height was added. This guides the individual to maintain a height range within a reasonable human body structure posture, improving balance control; the reward is defined as follows: (4).

[0040] In formula (4) It is the height of the root in the world coordinate system in the current state. This is the reference height when standing statically.

[0041] To achieve precise targeting of preset footsteps, a target footstep hit reward is introduced. This reward is calculated by comparing the spatial distance error between the robot's actual foot landing position and the corresponding target foot point. A higher positive reward is given to landing points with smaller errors, thereby encouraging the policy network to actively adjust stride length, leg swing trajectory, and gait rhythm to achieve precise foot landing. The reward is defined as follows: (5).

[0042] In formula (5) It is the distance between the next target foot and the nearest foot to it. It is the distance between the next target step point and the root, with a coefficient. , is an adjustable hyperparameter.

[0043] To improve the stability, energy efficiency, and control rationality of the network strategy, control clauses for action continuity and energy consumption are added to the reward function. Specifically, the difference in actions between the current time step and the previous time step is penalized to encourage the strategy network to output smooth and continuous control signals, avoiding sudden joint angle changes or control jumps. In addition, penalties are added for unreasonable collisions such as collisions between the robot's leg joints and base joints and the ground, and collisions between the robot's hip joints and knee joints, thereby ensuring that the robot is as stable and safe as possible during the learning process. Specifically, in equation (1) The definition is as follows: (6), where, It represents the action of the i-th joint in its current state and the action at the previous moment.

[0044] In formula (1) The definition is as follows: (7), among which, These are the torque output at the current moment and the torque output at the previous moment for the i-th joint, respectively.

[0045] In formula (1) The definition is as follows: (8), among which, It refers to the impact force on the knee and hip joints.

[0046] Step 3: Train the policy network.

[0047] Based on the aforementioned reward function, the constructed policy network is trained using the Mujoco simulation platform, and the PPO (Proximal Policy Optimization) algorithm is used to update the policy network. Before training, the robot's posture (e.g., semi-sitting) and terrain are initialized, with different step sequences preset for different terrains. During training, the robot's body state and step information for the next two moments in the current time, along with periodic signals, are collected as inputs to the policy network. The policy network outputs the positions of 12 target joints. The reward function is used to evaluate the output results of the policy network, measuring the quality of the action or behavior outcome. The PPO algorithm updates the policy network parameters. During training, the network policy continuously tries different actions, providing corresponding numerical feedback based on the reward function. The policy network gradually adjusts its parameters through optimization iterations, enabling its output actions to obtain higher cumulative rewards. Training continues until the trained network policy can stably complete the preset step sequences in the test environment, maintain balance on different terrains, and avoid frequent falls, achieving the set performance indicators. At this point, training is complete and a usable network policy is obtained. The typical design iteration count is 8000-10000 rounds.

[0048] In the simulation environment, the PD controller (Proportional-Derivative Controller) is used to calculate the actual control torque input reward function that needs to be applied to the robot's driven joints based on the difference between the desired joint angle (policy network output + initial angle) and the actual joint angle.

[0049] In the simulation environment, the entity state, the step information for the next two moments in the future, and the periodic signal can be obtained directly through the relevant API.

[0050] The body state includes joint angles and their velocities (12D), trunk posture (6D), and historical actions. The joints refer to the drive joints of the robot's legs, specifically the hip joint (pitch, yaw, roll), knee joint (pitch), and ankle joint (pitch, roll). The 12D refers to the state vectors of the robot's 12 leg motion quantities, each with joint angles and angular velocities, resulting in 12 dimensions for both joint angles and angular velocities. The trunk posture includes the robot's three-dimensional position (x, y, z) and three-dimensional posture (roll, pitch, yaw), totaling 6 dimensions. The historical actions are the joint target positions output by the policy network at the previous moment, obtained by recording the control commands or policy network output actions from the previous moment. The footstep information is the future step point data from a preset target footstep sequence.

[0051] The periodic signal refers to phase and time information. (2D) is used to encode gait phase, ensuring continuous and smooth phase information. φ is the normalized progress of a gait cycle, typically ranging from φ ∈ [0,1) (or [0,1], wrapping back to 0 at 1), where φ=0 indicates the start of a gait cycle, and φ=1 (or 0 after wrapping back) indicates the end of the cycle and the start of the next cycle. Phase is used to map the "time / gait progress" to a normalized parameter.

[0052] This embodiment uses a sliding window mechanism to collect step information for the next two moments from the current moment. Since the preset target step sequence contains the three-dimensional coordinates and one-dimensional orientation of several steps, but the policy network receives the target step information for the next two steps (relative to the current step), a dynamic sliding method needs to be designed. When the robot hits the next step, the window slides forward. Assume T1 and T2 are the index values ​​of the target step sequence. Initially, T1=0 and T2=1, corresponding to the first and second steps in the sequence (the index in the program starts from 0). When the robot's step hits T1, the index is updated: new T1 = original T2, and new T2 = original T2 + 1.

[0053] The initial terrain refers to calling different types of terrain in the simulation environment, setting a local coordinate system centered on the robot's body under its feet, and dividing it into an 8×8 grid. By calling the height query interface provided by the simulation environment, the ground height corresponding to each grid can be obtained, ultimately forming a 64-dimensional height vector. This data serves as the observation input, providing the geometric features of the local terrain under its feet for foot placement adaptation. The 8×8 grid height map (64D) under the robot's feet is used for local terrain adaptation at the foot placement point. Then, different foot sequence sequences are planned for different types of terrain. Each foot target consists of a three-dimensional position and a yaw angle (heading), used to describe the desired foot placement position and the corresponding torso orientation at that moment. The foot sequence is arranged in a temporal format, gradually guiding the robot to complete the global walking task.

[0054] The preset footstep sequences are all obtained from existing technologies. Different terrains have different footstep sequences, specifically divided into the following (1)-(3).

[0055] (1) In unobstructed flat terrain, the foot sequence is generated in a simple alternating manner: starting from the projection of the robot's root on the ground, the left and right foot landing points are arranged at intervals along the front or back direction. The step length (i.e. the distance between the left and right heels in the double support phase) and the distance between the left and right feet are fixed values ​​and are set for different robot models.

[0056] (2) For terrain containing obstacles, the foot planner in the Humanoid Navigation ROS package is used to automatically generate a curved path that can bypass the obstacles by adding random obstacles to the blank occupied grid map and setting the target pose (x, y, θ). The planner is based on a search strategy and can generate a series of feasible two-dimensional bipedal walking foot points given the start and end points.

[0057] (3) For complex terrain containing gaps, the foot sequence is generated to avoid gap areas based on the terrain elevation map, ensuring that the support points of the left and right feet always fall within the walkable area.

[0058] During training, once the current foot hits the target (i.e., it is within the target position tolerance range and maintains stable contact for a certain period of time), the foot information input strategy network for the next two moments is updated, thereby completing the step-by-step tracking and execution of the entire foot sequence; finally, under the guidance of the foot placement provided by the preset foot sequence, cross-terrain walking is achieved.

[0059] Step 4: Use the trained policy network for humanoid robot motion control, such as... Figure 2 As shown, the robot can jump on rocks, avoid obstacles, cross gaps, and walk on rough terrain with a stable and natural gait.

Claims

1. A footprint planning based motion control method for humanoid robots, characterized in that, The specific steps include: Step 1: Constructing a policy network The policy network uses a long short-term memory neural network structure, including four layers, namely an input layer, two serial hidden layers, and a linear output layer; Step 2: Designing a reward function The reward function is shown in formula (1): （1）； In formula (1) to encourage the robot to have correct ground contact force in the corresponding phase stage; to encourage the robot to accurately land the foot near the target point; to encourage the robot to maintain a reasonable speed in the swing and support stages; to encourage the robot to maintain the vertical height at the value of the standard posture; to punish the difference between the current action and the action at the last time; to punish the energy consumption; to punish the unreasonable collision; , , , , , , are weight coefficients; Step 3: Training the policy network Based on the above reward function, the constructed policy network is trained using the mujoco simulation platform, and the PPO algorithm is used to update the policy network; first, the robot posture and terrain are initialized, and different terrains are pre-set with different footstep sequences; the robot body state and the footstep information at the current time and the future two time points and the periodic signal are collected as the input of the policy network, the policy network outputs 12 target action joint positions, the reward function is used to evaluate the policy network output results, measure the advantages and disadvantages of the action or behavior results, and the PPO algorithm is used to update the policy network; Step 4: Using the trained policy network to control the motion of the humanoid robot.

2. The footprint planning based humanoid robot motion control method according to claim 1, wherein, In formula (1) and To encourage the robot to form a symmetrical gait with a clear support-swing rhythm in training, To encourage the supporting foot to have sufficient ground contact force in the support phase to ensure the stability of the robot during single-foot support or double-foot support; is defined as follows: （2）； wherein, and Fleft and Fright are the contact forces of the left and right feet with the ground in the current state, respectively; , φleft and φright are phase functions used to adjust the contact forces of the left and right feet in the support phase, respectively. To encourage the swing foot to maintain a reasonable swing speed forward during the swing phase, and avoid early contact with the ground, The following definitions apply: (3) where, and are the velocities of the left and right foot respectively in the current state; , are phase functions used to adjust the velocities of the left and right foot in the swing phase.

3. The footprint plan-based humanoid robot motion control method according to claim 1, wherein, In formula (1) are defined as follows: (4) is the height of the root in the world coordinate system in the current state, is the reference height in the static standing state.

4. The footprint plan-based humanoid robot motion control method according to claim 1, wherein, In formula (1) are defined as follows: （5） wherein is the distance between the next target footstep point and the distance between the next target footstep point and the root, is the distance between the next target footstep point and the distance between the next target footstep point and the root, is a hyperparameter.

5. The footprint plan-based humanoid robot motion control method according to claim 1, wherein, In formula (1) (6) wherein, is the motion of the ith joint at the current state and the motion at the previous time instant; （7） wherein, are the current and previous time output torques of the i-th joint, respectively. （8） wherein, is the impact force experienced by the knee and hip joints.

6. The footprint plan-based humanoid robot motion control method according to claim 1, wherein, The body state includes joint angle and its speed, torso posture and historical action, the joint refers to the driving joint of the robot leg, specifically including hip joint, knee joint and ankle joint; the torso posture includes the three-dimensional position and three-dimensional posture of the robot torso; the historical action is the joint target position output by the policy network at the last time; the footstep information is the future step point data in the pre-set target footstep sequence; the periodic signal refers to the phase time information.

7. The footprint planning based humanoid robot motion control method according to claim 1, wherein, During training, the window sliding mechanism is used to collect the footstep information at the current time and the future two time points.

Images