A humanoid robot lower limb walking control method and device

Abstract

The application relates to a humanoid robot lower limb walking control method and device, belonging to the technical field of robot control, comprising the following steps: acquiring state information and environment information of the robot; inputting the state information and the environment information into a policy network based on a self-attention structure, outputting a joint action sequence in multiple future steps, weighting and fusing candidate actions at the same future time of the outputted overlapping action sequence, and generating smooth and coherent control instructions; inputting a state transition segment of the robot into an adversarial prior network, outputting a style score and mapping the style score into a style reward; combining a task completion degree reward and the style reward to construct a comprehensive reward function, and optimizing the policy network; and deploying the optimized policy network to a robot controller to realize lower limb walking control. The application can effectively improve the naturalness, stability and environment adaptability of walking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a humanoid robot lower limb walking control method and device. BACKGROUND

[0002] The existing humanoid robot walking control method mainly has the following shortcomings: the traditional trajectory planning and control method needs a large amount of manual design of gait trajectory, and is difficult to cope with complex or unknown environment, and lacks self-adaptability and robustness.

[0003] The deep reinforcement learning method can realize walking through interaction learning, but if only relying on task reward, it is easy to generate asymmetric and unnatural gait, and the action jitter is obvious, and it lacks robustness to disturbance.

[0004] The single-step prediction strategy has the problem of error accumulation, which easily leads to unstable or invalid walking. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a humanoid robot lower limb walking control method and device.

[0006] The technical scheme adopted by the present application to solve the above technical problems is: a humanoid robot lower limb walking control method, comprising the following steps:

[0007] S1, obtaining state information and environment information of the robot;

[0008] S2, inputting the state information and environment information into a policy network based on a self-attention structure, and outputting a joint action sequence in multiple future steps;

[0009] S3, weighting and fusing the candidate actions at the same future time in the overlapping action sequence output in step S2 to generate smooth and coherent control instructions;

[0010] S4, inputting the state transition segment of the robot into an adversarial prior network, outputting a style score and mapping it into a style reward;

[0011] S5, combining the task completion reward and the style reward to construct a comprehensive reward function, and optimizing the policy network;

[0012] S6, deploying the optimized policy network to the robot controller to realize lower limb walking control.

[0013] Further, in step S2, the policy network adopts a Transformer structure based on a self-attention mechanism to encode the historical state sequence and output a multi-step action block sequence.

[0014] Further, in step S3, the weighted fusion adopts an exponential weighted average method of a time decay factor and a state adaptive factor.

[0015] Further, in step S4, the adversarial prior network is trained with high-quality reference motion data as positive samples and robot-generated trajectories as negative samples, adopts a least squares adversarial loss, and introduces gradient regularization on expert samples.

[0016] Further, in step S5, the reward function is a weighted sum of task completion reward and style reward, the style reward is derived from the output score of the adversarial prior network, and the task completion reward includes speed, posture stability, energy consumption and contact stability indicators.

[0017] Further, in step S5, the optimization of the policy network adopts a reinforcement learning algorithm based on policy gradient, including any one of PPO, SAC or TRPO.

[0018] The application also provides a device for implementing the above-mentioned humanoid robot lower limb walking control method, which comprises a thigh segment, a calf segment and a foot segment, the thigh segment is connected with the torso of the humanoid robot through a hip joint, the thigh segment and the calf segment are connected through a knee joint, and the calf segment and the foot segment are connected through an ankle joint, motor drivers and angle sensors are arranged in the hip joint, the knee joint and the ankle joint, a force sensor is arranged on the sole of the foot segment, and the device further comprises a robot controller connected with the motor drivers, the angle sensors and the force sensor.

[0019] The application has the following beneficial effects: 1. The application proposes a sequence action block prediction policy network based on Transformer, which expands the traditional single-step action output to multi-step trajectory block output, and adopts time decay and state adaptive exponential weighted fusion for overlapping blocks in the reasoning stage, thereby effectively suppressing long-time error accumulation and action jitter while maintaining real-time response, and greatly improving the stability and continuity of gait.

[0020] 2. The application constructs an adversarial prior network, uses high-quality reference motion data as true samples and robot-generated trajectories as false samples, outputs a style score by discriminating the state transition segment, and uses the style score as a style reward and a task completion reward for joint optimization, thereby significantly improving the coordination, symmetry and humanoid nature of the generated gait.

[0021] 3. The application integrates the style reward and the task completion reward output by the adversarial prior network into a total reward signal, and uses an existing mature policy gradient-based reinforcement learning algorithm (such as PPO, SAC, etc.) to optimize the policy network, thereby effectively improving the training sample efficiency while ensuring the stability of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0022] Fig. 1 is the overall framework diagram of the present application.

[0023] Fig. 2 is the training deployment flowchart of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more; the terms "upper", "lower", "left", "right", "inner", "outer", "front end", "rear end", "head", "tail" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" and the like are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0025] Please refer to Figs. 1-2 A humanoid robot lower limb walking control method, comprising the following steps:

[0026] Step 1, perception input: collect robot state information and environment information, the robot state information includes center of mass position, trunk posture, joint angle and angular velocity, foot position and contact information. The environment information includes terrain height.

[0027] Step 2, strategy network prediction based on self-attention structure: build a strategy network based on the Transformer structure, input the historical state sequence, output the joint control action sequence of the future several steps, and realize the global modeling of long time sequence action.

[0028] Step 3, overlapping action sequence weighted fusion: in the reasoning stage, the action sequence predicted by adjacent time steps produces multiple candidates at the same future time, and these candidates are exponentially weighted and fused according to the time decay and state adaptive factor to generate smooth and coherent action instructions, effectively suppressing long time sequence error accumulation and action jitter.

[0029] The action sequence weighted fusion adopts an exponential weighted average method:

[0030] ,

[0031] In the formula, is the predicted action of the tth step after fusion, is the predicted action of the ith action block at the tth step, is a weight coefficient, used to time-decay weighted fusion of the candidate actions of the overlapping block, to suppress long-term prediction error, is a decay coefficient, the decay coefficient is a scalar, controlling the decay of the weight with the time sequence of the block, i is the action block index.

[0032] Step four, against the prior network style constraint: build a discriminant network with high-quality reference motion as the true sample and robot-generated trajectory as the false sample, and output the style score of the state transition segment, and map it to the style reward.

[0033] The input of the anti-prior network is the local feature of the state transition segment , which includes root velocity, joint local rotation and angular velocity, end effector relative position, contact state, etc.

[0034] The network training adopts least square adversarial loss, taking high-quality reference motion segment as the true sample and strategy generation segment as the false sample. Gradient regularization is introduced on the expert sample to improve the stability of training.

[0035] The output of the anti-prior network is the style score , ranging from [0, 1], the closer to 1, the more similar to the reference motion. The style reward is obtained by linear mapping:

[0036]

[0037] In the formula, is the robot state of adjacent two frames, including root velocity, joint angle, angular velocity, end relative position, contact state; is the style reward, ranging from [0, 1].

[0038] Step five, reward function comprehensive optimization: the total reward is formed by combining the task completion degree reward and the style reward output by the anti-prior network, guiding the strategy network to generate actions that can complete the task and have natural style.

[0039] The comprehensive reward function is:

[0040]

[0041] In the formula, is the total reward at time t, is the task completion degree reward, is the style reward, and are weight coefficients, used to balance the task and style rewards. The weight is determined by experience setting and parameter search, and the training initial period , , can be appropriately fine-tuned according to the learning curve.

[0042] Step six, reinforcement learning policy optimization: a policy gradient-based reinforcement learning algorithm is used to train the policy network, such as proximal policy optimization (PPO), soft actor-critic (SAC), or trust region policy optimization (TRPO), to maximize the expected cumulative reward.

[0043] Step seven, simulation training and migration deployment: curriculum learning and domain randomization training are performed in a simulation environment, parallel training is performed on a simulation platform, curriculum learning strategies are used to gradually transition from simple terrain to complex terrain. Domain randomization includes dynamic parameters, friction coefficients, sensor noise, and delay disturbances. Before real deployment, the system identifies and calibrates the real robot dynamics parameters, and performs policy distillation and delay compensation to improve the real-time performance and stability of the policy.

[0044] The present application performs parallel training on a simulation platform, uses curriculum learning and domain randomization to improve robustness, performs system identification and parameter correction before real deployment, uses policy distillation and delay compensation to improve real-time performance, adds a safety constraint layer and a fall recovery mechanism to the robot controller, and further improves performance through online fine-tuning or residual learning when necessary.

[0045] Step eight, real deployment and safety layer: deploy the optimized policy on a real robot, add joint limits, torque saturation, contact hysteresis, and fall recovery mechanisms.

[0046] The present application also provides a device for implementing the above-mentioned humanoid robot lower limb walking control method, including a thigh segment, a calf segment, and a foot segment. The thigh segment is connected to the humanoid robot torso through a hip joint, the thigh segment and the calf segment are connected through a knee joint, and the calf segment and the foot segment are connected through an ankle joint. Motor drivers and angle sensors are provided in the hip joint, knee joint, and ankle joint. The sole of the foot segment is provided with a force sensor. The device further includes a robot controller connected to the motor drivers, angle sensors, and force sensors. The foot structure is provided with a non-slip pad layer and a buffer mechanism to improve contact stability. The controller is an embedded microprocessor running a microkernel real-time operating system and supports EtherCAT or CAN bus communication. The present embodiment communicates with the motor drivers and sensors through the EtherCAT bus, executes the above-mentioned humanoid robot lower limb walking control method, outputs smooth joint commands, and realizes natural and stable lower limb walking.

[0047] The humanoid robot trunk is provided with an inertial measurement unit (IMU) for acquiring the mass center and attitude information. Angle encoders are installed in the hip joint, knee joint and ankle joint for collecting angle and angular velocity. A six-dimensional force sensor is installed on the foot bottom for detecting ground contact force and friction information. The hip joint and knee joint are single-degree-of-freedom rotary joints, and the ankle joint is a two-degree-of-freedom joint allowing pitch and roll.

[0048] It should be noted that the above examples are only used to illustrate the present application, but the present application is not limited to the above examples, and any simple modification, equivalent change and modification of the above examples according to the technical essence of the present application all fall within the protection scope of the present application.

Claims

1. A humanoid robot lower leg walking control method characterized by comprising: The method comprises the following steps: S1, obtaining state information and environment information of the robot; S2, inputting the state information and the environment information into a policy network based on a self-attention structure to output a joint action sequence in multiple future steps; S3, weighting and fusing candidate actions at the same future time in the joint action sequence output in step S2: the joint action sequence predicted at adjacent time steps generates multiple candidate actions at the same future time, the multiple candidate actions are weighted and fused to generate smooth and coherent control instructions; S4, inputting a state transition segment of the robot into an adversarial prior network to output a style score and map it to a style reward; S5, combining a task completion reward and the style reward to construct a comprehensive reward function, and optimizing the policy network; S6, deploying the optimized policy network to a robot controller to realize lower limb walking control.

2. The walking control method of a lower leg of a humanoid robot according to claim 1, characterized by, In step S2, the policy network adopts a Transformer structure based on a self-attention mechanism to encode the historical state sequence and output a future multi-step action block sequence.

3. The walking control method of a lower leg of a humanoid robot according to claim 1, characterized by, In step S3, the weighted fusion adopts an exponential weighted average method of a time decay factor and a state adaptive factor.

4. The walking control method of a lower leg of a humanoid robot according to claim 1, characterized by, In step S4, the adversarial prior network is trained with high-quality reference motion data as positive samples and robot-generated trajectories as negative samples, adopts a least squares adversarial loss, and introduces gradient regularization on expert samples.

5. The walking control method of a lower leg of a humanoid robot according to claim 1, characterized by, In step S5, the reward function is a weighted sum of the task completion reward and the style reward, the style reward is derived from the output score of the adversarial prior network, and the task completion reward includes speed, posture stability, energy consumption, and contact stability indicators.

6. The walking control method of a lower leg of a humanoid robot according to claim 1, characterized by, In step S5, the optimization of the policy network adopts a reinforcement learning algorithm based on policy gradient, including any one of PPO, SAC or TRPO.

7. An apparatus for implementing the walking control method of the lower limbs of a humanoid robot according to any one of claims 1 to 6, characterized in that, The robot comprises a thigh segment, a lower leg segment and a foot segment, the thigh segment is connected with a humanoid robot torso through a hip joint, the thigh segment and the lower leg segment are connected through a knee joint, the lower leg segment and the foot segment are connected through an ankle joint, motor drivers and angle sensors are arranged in the hip joint, the knee joint and the ankle joint, a force sensor is arranged on the sole of the foot segment, and the robot further comprises a robot controller connected with the motor drivers, the angle sensors and the force sensor.

Images