A method for generating motion at multiple contact points in a humanoid robot

Abstract

This invention provides a method for generating multi-contact point motion in humanoid robots, comprising: describing the motion state of the end effector of a multi-contact point legged robot; constructing a mathematical description of the motion mode of the robot's end effector; converting the mathematical description into a reward function for reinforcement learning to train motor skills; and finally transferring the method to a physical robot. The advantages of this method are that it provides a comprehensive and accurate mathematical description of the end effector motion of a multi-contact point robot, constructs a continuous reward function to improve the continuity of the parameter space, and effectively guides the algorithm to converge quickly to the desired state, making it particularly suitable for generating motor skills based on reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of motion control for humanoid robots, and more particularly to a method for generating motion at multiple contact points in humanoid robots. Background Technology

[0002] In the field of humanoid robot motion control, robots control their state through contact and interaction with the environment via their end-effectors. This is a multi-contact motion. The state of the robot's end-effectors determines the way the robot interacts with the environment; contact between the feet and the ground is fundamental to common tasks such as walking and maintaining balance. Simultaneously, by utilizing the upper limbs, robots can expand their stability domain by contacting other planes, enabling them to navigate more complex terrains and achieve stronger overall performance. Examples include using both hands and feet to climb ladders, holding onto walls to ascend and descend stairs, and navigating narrow paths.

[0003] Control methods for legged robots can be divided into model-based control and learning-based data-driven control. Both methods have achieved good results. Data-driven control based on reinforcement learning is currently a hot topic. Using a deep neural network as the controller, the task of optimizing complex strategies is delegated to the algorithm during training. The network's nonlinear fitting ability and the training algorithm's autonomous exploration ability are utilized to find flexible solutions. During reinforcement learning, the robot continuously interacts with the environment in simulation, autonomously updating the network according to a preset high-level reward function to obtain a strategy that maximizes the reward. Currently, the motion strategies generated by reinforcement learning have been successfully deployed in quadrupedal, bipedal, and humanoid robots, possessing certain motion capabilities, such as walking, running, jumping, and disturbance resistance. Ren Ju (CN111360834A) et al. proposed a motion control method and system for humanoid robots based on deep reinforcement learning. In simulation, the current state of the humanoid robot is obtained, and the target angles of each joint of the humanoid robot are calculated and determined based on the current state using a preset deep reinforcement learning model. Then, a PD controller executes the following. Liu Rong et al. (CN112051735A) proposed a walking control method for humanoid robots, which performs offline gait planning to enable the humanoid robot to walk by tracking offline-generated joint motion trajectories. Zhang Guoliang et al. (CN103770111A) proposed a gait planning and synthesis method for humanoid robots, using polynomial functions to represent the trajectories of the robot's hip joint and swing leg. Based on constraints such as geometric constraints, maximum stride height of the swing leg, gait periodicity, collision effects, and hip joint motion, the gait is planned for both single-leg and double-leg support phases during walking. The planned gait is then synthesized, and the stability of the gait trajectory is determined based on the zero-point torque criterion. Overall, most of these motion algorithms only focus on the motion state of the robot's center of mass, without clearly specifying the motion state of the robot's end effector, or only specifying a portion of the end effector state, thus not fully utilizing the robot's performance.

[0004] Existing technologies can achieve tasks such as walking, balancing, and manipulating, but they are only designed for specific tasks faced by specific robots and are not universal methods for generating multi-contact point locomotion in legged robots. Summary of the Invention

[0005] To address the shortcomings of existing technologies in fully planning the end-effector state of humanoid robots, this invention proposes a multi-contact point motion generation method for humanoid robots.

[0006] To achieve the above objectives, the present invention provides a method for generating multi-contact point motion in a humanoid robot, comprising the following steps:

[0007] S1: Describe the motion state of the end effector of a multi-contact humanoid robot from two aspects: leg movement and arm operation;

[0008] S2: Based on the description in step (1), construct a mathematical description of the motion mode of the robot's limb end, and transform the mathematical description into a reward function in reinforcement learning;

[0009] S3: Build a reinforcement learning environment, design other reward items, and use reinforcement learning algorithms to generate motion control strategies for multi-contact humanoid robots.

[0010] S4: Transfer and deploy the motion control strategy generated in step (3) to the humanoid robot.

[0011] Further, step S1 specifically includes:

[0012] The end-effectors of humanoid robots include the arm end-effectors and the leg end-effectors. For all end-effectors, the motion process is divided into a support phase that is in stable contact with the support plane or support point and a swing phase that is suspended in the air. By combining the support phase and swing phase of each leg with different durations, arbitrary gait can be generated.

[0013] Further, step S2 specifically includes:

[0014] Based on the division of motion states into the support phase and the swing phase, this is reflected in the reward function of reinforcement learning; that is, the foot's movement speed is penalized during the support phase, and the contact force on the foot is penalized during the swing phase. Let the foot's movement speed and foot contact force each be a reward term, expressed as follows: and The deviation between the two rewards It is the degree of use of the left and right feet. Deviation from the original The sum of the products of can be expressed as follows:

[0015]

[0016] Activation level It is a binary function that switches between different phases, indicating whether the deviation is active in the current phase. An activation level of 0 means no penalty and the highest reward is obtained directly; an activation level of 1 means full penalty and the highest reward is obtained without bias; the activation level multiplied by the original reward is the real reward item collected as data at this moment, and the deviation level is the Euclidean distance between the observation and the target; for foot movement speed, the activation level is 1 in the support phase and 0 in the swing phase; for foot contact force, the activation level is 0 in the support phase and 1 in the swing phase.

[0017] Furthermore, phase switching can be performed using linear interpolation, tanh, sigmoid, or von Mises distribution functions.

[0018] Furthermore, step S3 specifically includes:

[0019] A parallel environment for reinforcement learning is built in simulation software to support simultaneous training of multiple limbs; a neural network is built as the robot's control strategy, and a proximal policy optimization algorithm is used to learn and optimize the motion control strategy; a body state reward term and a performance optimization reward term are designed, and the policy parameters are iteratively updated by maximizing the reward function, so that the robot's motion control strategy gradually approaches the optimal solution.

[0020] Furthermore, the body state reward items include linear velocity command tracking, angular velocity command tracking, posture stability, and height maintenance; the performance optimization reward items include optimizing foot contact force impact, smoothing joint torque, smoothing joint velocity, and reducing energy consumption.

[0021] Further, step S4 specifically includes:

[0022] After the motion control strategy training is completed, it is exported into a format that can be run on a humanoid robot, and then tested on an actual humanoid robot.

[0023] The beneficial effects of this invention are:

[0024] (1) The present invention proposes a method for complete planning of end-effector motion. Due to the accuracy and uniqueness of the description, the algorithm can stably converge to the desired motion state, accurately control the robot's motion behavior, and generate controllable and adjustable motion strategies.

[0025] (2) The multi-contact point motion generation method proposed in this invention has good scalability. For new end-effectors and new task combinations introduced by new mechanical structures, by adding a unified description of contact points and modifying reward terms, a new control strategy that meets expectations can be trained under the same framework.

[0026] (3) The present invention is theoretically applicable to various humanoid robot platforms, such as Wukong, Digit, HRP, COMAN, etc., and is universal for various common tasks. Attached Figure Description

[0027] Figure 1 This is a block diagram of a method for describing the multi-contact point motion generation of a humanoid robot according to the present invention;

[0028] Figure 2 This is a schematic diagram illustrating the bipedal walking gait of a humanoid robot, as described in this invention.

[0029] Figure 3 This is a schematic diagram illustrating the activation level of the bipedal speed and contact force reward items of a humanoid robot, as described in this invention.

[0030] Figure 4 This is a schematic diagram illustrating the phase signal input of the bipedal walking gait of a humanoid robot, as described in this invention.

[0031] Figure 5 This is a schematic diagram illustrating the test results of a humanoid robot as described in this invention. Detailed Implementation

[0032] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The objectives and effects of the present invention will become clearer as a result. The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0033] See Figure 1 The present invention provides a method for generating multi-contact point motion of a humanoid robot, which is implemented through the following steps:

[0034] S1: Describes the motion state of the extremities of a multi-contact humanoid robot (such as a legged robot).

[0035] No matter how complex an organism's gait is, for the extremities (including the feet and arms), the movement process can be divided into a support phase with stable contact with a support plane or support point and a swing phase with suspension. Combining the support and swing phases of each leg with different durations can generate any gait. During the support phase, the extremity should be in complete contact with the ground without slipping, so the expected velocity is 0; during the swing phase, the extremity should be completely suspended in the air without rubbing against the ground, so the expected contact force is 0. Based on the stability of ground contact and movement requirements, the duration and sequence of the support and swing phases of each limb are automatically adjusted to generate movement strategies adapted to different gaits and tasks. This method can describe the movement states of all extremities comprehensively and accurately, including the only correct gait and excluding all other unsuitable gaits. Examples include walking, running, and jumping of the legs, and moving the arms to a target position and supporting the body.

[0036] S2: Construct a mathematical description of the movement of the robot's limb end effector and transform it into a reward function.

[0037] The key to reinforcement learning lies in comprehensively and accurately describing the desired action and reflecting it in the reward function, establishing a suitable gradient in the parameter space, so that the trajectory and state of the desired action are local extrema in the "parameter-reward" space, and there are no second local extrema nearby. Take the bipedal walking of a humanoid robot as an example. The key to walking is alternating steps. The gait signal for the alternating steps of a humanoid robot is set as follows: one cycle is defined as the time from the first lift of a single foot to the next lift, with the support phase and swing phase each accounting for 50%. The cycles of the left and right feet are the same, with a phase difference of half the cycle. One foot's support phase corresponds to the swing phase of the other foot; there is no air phase. Figure 2 As shown, blue represents the left foot and orange represents the right foot.

[0038] Reflected in the reward function, this means: during the support phase, the foot's movement speed is penalized, and during the swing phase, the contact force on the foot is penalized. Let the foot's movement speed and contact force be the reward terms, expressed as follows: and The deviation between the two rewards It is the degree of use of the left and right feet. Deviation from the original The sum of the products of .

[0039]

[0040] Enable degree function This is a binary function that switches between different phases, indicating whether the deviation is active in the current phase. An activation level of 0 means no penalty and the highest reward is directly obtained; an activation level of 1 means full penalty, and the highest reward is obtained without bias. The activation level multiplied by the original reward is the actual reward term collected as data at this moment, and the deviation level is the Euclidean distance between the observation and the target. For foot movement velocity, the activation level is 1 in the support phase and 0 in the swing phase; for foot contact force, the activation level is 0 in the support phase and 1 in the swing phase. Periodic reward functions are these reward functions that vary according to phase within the stepping cycle.

[0041] Both feet can be viewed as copies of two single feet, differing only in phase. The calculation method using the same function is applied, therefore each foot has a time variable, differing by half a period, according to the time variable... Each update method is updated separately, using and express.

[0042] During phase switching, the continuity of the activation level should be maintained to make the "parameter-reward" space more continuous, which is beneficial for calculating gradients during updates. Phase switching can use any function. Linear interpolation, tanh, sigmoid, and von Mises distributions all provide sufficiently good results.

[0043] In the calculation, the deviation between the foot movement speed and the foot contact force reward is as follows:

[0044]

[0045]

[0046] in, The reward deviation, representing the foot movement speed, This indicates the reward deviation of the foot contact force. This indicates the activation level of the left foot movement speed bonus. This indicates the activation level of the left foot contact force reward. Indicates the speed of movement of the left foot. Indicates the contact force of the left foot. This indicates the activation level of the right foot movement speed bonus. This indicates the activation level of the right foot contact force reward. Indicates the movement speed of the right foot. This represents the contact force of the right foot; during walking, the phase difference between the left and right feet is half, and the movement speed is opposite to the activation function of the contact force. The graph is symmetrical. Figure 2 The extent to which the incentives for foot speed and contact force are utilized is clearly demonstrated in the visual display.

[0047] The reward function mainly focuses on the trade-off between achieving stable contact and efficient swing, thereby improving the overall efficiency and adaptability of the movement. This invention uses the generalized Cauchy kernel function as the form of the gait reward function, as shown in formula (4). The foot movement speed and foot contact force are the reward terms, as shown in formulas (5) and (6), respectively.

[0048]

[0049]

[0050]

[0051] in, Represents the generalized Cauchy kernel function. Indicates the intensity of the reward. Indicates the reward decay order. This represents the normalization coefficient of a physical quantity; it provides the phase signals of both feet as input to the control strategy, enabling the strategy to establish a relationship between the phase signals and reward acquisition during training. Ultimately, changing the frequency and value of the phase signals results in corresponding changes in gait, increasing the controllability of the strategy. The phase signal of each foot consists of a sine function and a cosine function, such as... Figure 3 and Figure 4 As shown.

[0052] S3: Build a reinforcement learning environment, design other rewards, and use reinforcement learning to generate motion control strategies for humanoid robots (such as legged robots).

[0053] To implement the motion control strategy described above, this invention establishes a parallel environment in simulation software to support simultaneous training of multiple limbs. A neural network is constructed as the robot's control strategy, and the Proximal Policy Optimization (PPO) algorithm is used to learn and optimize the control strategy. By iteratively updating the policy parameters by maximizing the reward function, the robot's motion strategy gradually approaches the optimal solution.

[0054] In this example, the robot's task is to walk on various complex terrains. To ensure the completion of the task, a series of reward functions were designed. In addition to gait rewards, this example added body state rewards and performance optimization rewards. Training was completed by waiting for the rewards to converge. In addition to the generalized Cauchy kernel function mentioned in step S2, this example also used the Gaussian kernel function as part of the reward term, as shown in formula (7).

[0055]

[0056] in, This indicates some of the reward items; the physical condition reward items include linear velocity command tracking. Angular velocity command tracking Stable posture and maintain height The order is as follows:

[0057]

[0058]

[0059]

[0060]

[0061] The asterisk (*) in the upper right corner represents the expected value of a certain physical quantity. Represents the linear velocity at time t. Represents the angular velocity at time t. Represents the vertical distance between the body and the ground. It is the value in the lower right corner of the body system rotation matrix. This indicates the degree of deviation between the robot's body-system Z-axis and the world-system Z-axis.

[0062] Performance optimization bonuses include improved impact force on the soles of the feet. Smooth joint torque Smooth joint velocity and reduce energy consumption The order is as follows:

[0063]

[0064]

[0065]

[0066]

[0067] in, Represents the foot contact force at time t. Represents the robot's weight. Represents the joint torque at time t. This represents the joint velocity at time t.

[0068] The robot used in this example is the humanoid robot "Wukong-IV", which is 1.4m tall in a standing posture, weighs 45kg, and has 21 degrees of freedom.

[0069] S4: Policy migration and deployment to physical robots.

[0070] After the policy training was completed, it was exported to a format executable on the robot and tested on the actual robot "Wukong-IV". Given a step frequency of 1.5 Hz, the results are as follows: Figure 5 As shown. Based on intuitive visual effects and a comparison of the humanoid robot's knee movement frequency, foot lift height, ground force, and gait signals, "Wukong-IV" is able to follow the gait frequency.

[0071] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for generating multi-contact point motion in a humanoid robot, characterized in that, Includes the following steps: S1: The motion state of the end-effectors of a multi-contact humanoid robot is described from two aspects: leg movement and arm operation. The end-effectors of the humanoid robot include the arm end-effectors and the leg end-effectors. For all end-effectors, the motion process is divided into a support phase with stable contact with the support plane or support point and a swing phase with suspension. Arbitrary gait is generated by combining the support phase and swing phase of each leg with different durations. S2: Based on the description in step S1, construct a mathematical description of the motion mode of the end of the robot limb, and convert the mathematical description into a reward function in reinforcement learning; including: according to the motion state division of support phase and swing phase, the reward function in reinforcement learning is reacted; that is, the moving speed of the foot is punished in the support phase, and the contact force on the foot is punished in the swing phase; let the foot moving speed and the foot contact force be reward items, respectively represented as And The deviation degree of the two rewards Is the product sum of the left and right foot activation degree And the original deviation degree , which has the following expression: ； enablement is a binary function that switches between phases, indicating whether the deviation is in the active state at the current phase, enablement of 0 means no punishment, and the highest reward is obtained directly; enablement of 1 means complete punishment, and the highest reward is obtained without deviation; the enablement multiplied by the original reward is the real reward item collected as data at this moment, and the deviation is the Euclidean distance between the observation and the target; for the foot movement speed, the enablement is 1 in the support phase, and the enablement is 0 in the swing phase; for the foot contact force, the enablement is 0 in the support phase, and the enablement is 1 in the swing phase; S3: Build a reinforcement learning environment, design other reward items, and use reinforcement learning algorithms to generate motion control strategies for multi-contact humanoid robots. S4: Migrate and deploy the motion control strategy generated in step S3 to the humanoid robot.

2. The human-imitative robot multi-contact point motion generation method according to claim 1, characterized by, Phase switching can be performed using linear interpolation, tanh, sigmoid, or von Mises distribution function.

3. The humanoids robot multi-contact motion generation method according to claim 1, wherein, Step S3 specifically includes: A parallel environment for reinforcement learning is built in simulation software to support simultaneous training of multiple limbs; a neural network is built as the robot's control strategy, and a proximal policy optimization algorithm is used to learn and optimize the motion control strategy; a body state reward term and a performance optimization reward term are designed, and the policy parameters are iteratively updated by maximizing the reward function, so that the robot's motion control strategy gradually approaches the optimal solution.

4. The human-imitating robot multi-contact point motion generation method according to claim 3, characterized by, The physical state reward items include linear velocity command tracking, angular velocity command tracking, posture stability, and height maintenance; the performance optimization reward items include optimizing foot contact force impact, smoothing joint torque, smoothing joint velocity, and reducing energy consumption.

5. The method for generating multi-contact point motion of a humanoid robot according to claim 1, characterized in that, Step S4 specifically includes: After the motion control strategy training is completed, it is exported into a format that can be run on a humanoid robot, and then tested on an actual humanoid robot.

Images