Motion control method for quadruped robot with damaged legs

By combining CPG controller and deep reinforcement learning in the motion control of quadruped robots, the fuselage balance problem in the event of sudden failure is solved, and the adaptive movement of quadruped robots in the case of leg damage is achieved. Environmental adaptation improves robustness and adaptability.

CN118915802APending Publication Date: 2024-11-08TONGJI UNIV
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202410971117.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing four-legged robot motion control method cannot effectively deal with single-leg failure when facing sudden failures of leg motors or joints, resulting in damage to the fuselage balance and lacks robustness and adaptability.

Method used

Using a two-layer framework based on CPG controller and deep reinforcement learning, high-level reinforcement learning generates CPG parameters, underlying reinforcement learning fine-tunes joint trajectory, combined with Hopf oscillator and PPO reinforcement learning algorithm, a strategy selection state machine is built to realize the four-legged robot in a sudden burst Adaptive switching and environmental adaptation in the event of a failure.

Benefits of technology

It improves the movement robustness and environmental adaptability of the four-legged robot, can maintain balance and effectively move in the event of sudden leg damage, adapt to complex terrain, and enhances the ability to respond to sudden failures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915802A_ABST
    Figure CN118915802A_ABST
Patent Text Reader

Abstract

The invention relates to a motion control method for a quadruped robot with damaged legs. A CPG rhythm controller is constructed for the quadruped robot; when the legs of the quadruped robot break down, switching between the normal movement strategy of the quadruped robot and the movement strategy of the quadruped robot with the damaged legs is carried out; wherein a quadruped robot motion strategy adopts a double-layer reinforcement learning framework: high-layer reinforcement learning generates CPG parameters according to a current robot state and a reference command, and generates a joint reference trajectory; carrying out fine adjustment on the generated reference joint track through bottom reinforcement learning to obtain a target joint angle; and accumulating the joint reference trajectory and the target joint angle, and further calculating to obtain a current robot joint torque instruction. According to the method, the robot can simulate normal movement and gait properties when the legs are injured, CPG parameters and joint angles are adjusted through the double-layer PPO reinforcement learning model, and the problems of walking robustness and environmental adaptability of the quadruped robot with the injured legs are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot control, and in particular to a motion control method for a quadruped robot with damaged legs based on a CPG controller and deep reinforcement learning. Background Art

[0002] The gait control method of quadruped robots is a core issue in the research field of quadruped robots. The gait control of quadruped robots with damaged legs requires a more complex control scheme due to the changes in their movement rhythm and posture center of gravity. CPG (central pattern generator) is a biological neural circuit that produces rhythmic movement behavior of animals. It is composed of a series of neural oscillators. It is a complex distributed neural network composed of neural oscillators and multiple reflex loop systems. It is one of the most classic control methods in the field of gait control of quadruped robots. At the same time, reinforcement learning algorithms have developed rapidly in recent years. They have shown strong robustness and generalization capabilities in the field of robot control. Many foreign research institutions have been able to use reinforcement learning algorithms to train quadruped robot movement strategies that can move autonomously in complex unstructured environments.

[0003] However, both the CPG solution and the reinforcement learning solution have some limitations. The CPG solution often requires planning the foot position coordinate trajectory in the Cartesian coordinate system or the joint angle change trajectory in the joint space, which makes the method lack robustness to sudden internal and external disturbances. Reinforcement learning requires continuous attempts in network architecture design and reward function debugging. In the absence of prior trajectories, it often takes a very long time of training to obtain a robust strategy that can be generalized to different situations to achieve the desired effect.

[0004] Chinese patent application CN202010095442.4 discloses a quadruped robot gait control method based on reinforcement learning and CPG controller for a normal quadruped robot, which calculates the foot end motion trajectory of the quadruped robot and then controls the robot motion. However, such a strategy is only effective for normal quadruped robots. Once the leg motor or joint fails suddenly, the strategy will no longer guarantee the normal movement of the robot. When the current mainstream CPG and reinforcement learning models deal with the problem of quadruped robot motion control, the high-level solutions all design algorithms and reward functions for the overall model of the four legs, and cannot cope with sudden single-leg failures. When a single-leg failure occurs, the original reinforcement learning state-action mapping cannot be correctly executed due to the joint action, and the balance of the entire body will be destroyed. Therefore, it is necessary to design a quadruped robot motion control algorithm to deal with leg damage. Summary of the invention

[0005] The purpose of the present invention is to overcome the defect that the above-mentioned prior art cannot cope with sudden failures of leg motors or joints and to provide a motion control method for a quadruped robot with damaged legs based on a CPG controller and deep reinforcement learning.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A motion control method for a quadruped robot with damaged legs, wherein the method constructs a CPG rhythm controller according to a quadruped robot model;

[0008] When any leg of the quadruped robot fails, the normal quadruped robot motion strategy of the quadruped robot and the leg-damaged quadruped robot motion strategy are switched to control the motion of the quadruped robot;

[0009] The motion strategies of the normal quadruped robot and the quadruped robot with damaged legs adopt a two-layer reinforcement learning framework:

[0010] High-level reinforcement learning generates CPG parameters based on the current robot state and reference commands, and the CPG parameters generate joint reference trajectories;

[0011] The underlying reinforcement learning fine-tunes the reference joint trajectory generated by CPG to obtain the target joint angle at each moment;

[0012] The joint reference trajectory and the target joint angle are accumulated to obtain the current robot joint torque command.

[0013] As an optimal technical solution, the CPG rhythm controller of the quadruped robot adopts a Hopf oscillator, and an oscillator is assigned to each leg of the quadruped robot. The parameters of each oscillator reflect the rhythmic motion information of the corresponding leg. The oscillator generates a periodic oscillation signal at the leg joint and obtains the phase difference between the undamaged diagonal legs and between the two legs on the undamaged side when one leg is damaged, thereby providing the quadruped robot motion prior for the underlying reinforcement learning fine-tuning.

[0014] As a preferred technical solution, in the two-layer reinforcement learning framework, both the high-level and low-level strategies adopt the PPO reinforcement learning algorithm:

[0015] The high-level reinforcement learning policy network takes as input privileged information under the simulator, robot proprioceptive observations, external reference speed instructions, and CPG parameters of the previous step, and outputs target CPG parameters.

[0016] The target CPG parameters output by high-level reinforcement learning are mapped to construct the coarse joint space trajectory of each undamaged leg;

[0017] The underlying reinforcement learning strategy network inputs the observation values ​​of the robot's proprioception, including the robot's current speed, posture angle, angles and angular velocities of each joint, and the contact state between the foot and the ground; and outputs the residual angle of each joint;

[0018] The residual angle output by the bottom-level reinforcement learning is added to the coarse joint space trajectory formed by the target CPG parameters output by the high-level reinforcement learning to obtain the final joint target trajectory.

[0019] As a preferred technical solution, in the input of the high-level reinforcement learning strategy network, the privileged information under the simulator includes the current speed of the robot, the mass of the robot, the ground friction coefficient, and the local elevation information of the ground near the robot; the robot proprioception includes the current speed of the robot, the robot posture angle, the angle and angular velocity of each joint, and the contact state between the foot end and the ground;

[0020] The high-level reinforcement learning strategy network output target CPG parameters of the normal quadruped robot motion strategy include: thigh swing amplitude, period, duty cycle, and gait type;

[0021] The high-level reinforcement learning strategy network outputs the target CPG parameters of the leg-impaired quadruped robot's motion strategy, including thigh swing amplitude, period, duty cycle, and the relative phase difference of the remaining three legs.

[0022] As a preferred technical solution, the reward function of the high-level reinforcement learning strategy network is used to achieve task rewards including speed tracking and terrain traversal:

[0023] The bonus items for speed tracking are:

[0024]

[0025] Where: Indicates the speed command in the x and y directions; v xy represents the current velocity of the robot in the x and y directions; σ is a shaping scalar;

[0026] The rewards for terrain traversal are:

[0027] r2=n success

[0028] Where n success Indicates the number of agents that successfully traverse the terrain in the simulation;

[0029] The reward function of the underlying reinforcement learning policy network is used to maintain the body balance:

[0030]

[0031] Where: v zIndicates the robot's speed in the z direction; ω xy Indicates the angular velocity of the robot in the rolling and pitching directions.

[0032] As a preferred technical solution, the reward functions of the high-level reinforcement learning strategy and the low-level reinforcement learning strategy also include regularized reward items including joint acceleration limit, action change rate, and smoothness, as follows:

[0033]

[0034] In the formula, represents the joint angular acceleration;

[0035]

[0036] In the formula, a t represents the motor joint action at the current time step, a t-1 Represents the motor joint action of the previous time step;

[0037]

[0038] In the formula, a t-2 represents the motor joint action two time steps ago.

[0039] As an optimal technical solution, the two-layer reinforcement learning framework adopts a teacher-student strategy distillation framework to distill the high-level reinforcement learning model and the policy network of the high-level reinforcement learning model; uses historical observations and action sequences of past set frames to replace prior observations in the simulator; and obtains a hierarchical learning network deployed in the real world through supervised learning.

[0040] As a preferred technical solution, during the model training process of the double-layer reinforcement learning framework, a course training form is adopted, and the robot is alternately trained in multiple scenarios to enhance the robot's stability and adaptability to terrain. The scenarios include steps, slopes and unstructured roads.

[0041] As a preferred technical solution, the method step inputs each joint angle in the final joint target trajectory into a PD controller to obtain a joint torque, and uses the joint torque as a gait control signal of the quadruped robot to drive the quadruped robot.

[0042] As a preferred technical solution, the method constructs a strategy selection state machine based on a neural network, and the strategy selection state machine adaptively switches the motion strategy of the quadruped robot according to specific situations.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1) This paper uses a hierarchical framework to effectively combine the CPG controller with the PPO reinforcement learning model to learn the motion strategies of normal quadruped robots and quadruped robots with damaged legs on flat terrain. The high-level reinforcement learning generates CPG parameters based on the current robot state and reference commands. The CPG parameters will generate joint reference trajectories via oscillators. The bottom-level reinforcement learning further fine-tunes the reference joint trajectories generated by CPG, making the motion of the quadruped robot more robust.

[0045] 2) In order to cope with the exploration of complex terrain, the present invention uses a course learning scheme to set courses from low difficulty to high difficulty for scenes such as slopes, steps, and unstructured terrain under the simulator, so as to further fine-tune the hierarchical network parameters. Thanks to the ability of the underlying reinforcement learning to adjust the reference joint trajectory pre-generated by CPG, the problem of environmental adaptability walking of the quadruped robot with damaged legs is solved.

[0046] 3) Construct a strategy switching state machine to deal with emergencies. When any leg of the quadruped robot is damaged, the strategy will autonomously switch to the corresponding strategy when the leg is damaged. Our solution enables the quadruped robot to cope with sudden leg damage. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the motion control method of a quadruped robot with damaged legs according to the present invention;

[0048] Figure 2 This is a CPG network structure diagram of the main control network for controlling the three-leg hip joints of the present invention. DETAILED DESCRIPTION

[0049] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0050] Example 1

[0051] The present invention further studies the movement mechanism of quadrupeds, considers the movement scheme of a quadruped robot with damaged legs, and proposes a gait control method for a quadruped robot with damaged legs based on a CPG controller and deep reinforcement learning, constructs a CPG rhythm controller for the quadruped robot model with damaged legs; trains the high-level reinforcement learning model of the quadruped robot to optimize the parameters of the CPG oscillator; maps the signal output by the CPG to construct the joint angle change trajectory of each undamaged leg; trains the low-level reinforcement learning model of the quadruped robot, and fine-tunes the joint angle trajectory output by the CPG according to the current robot state. The fine-tuned joint angle will be input into the PD controller to obtain the joint torque, and the joint torque will be used as the gait control signal of the quadruped robot to drive the quadruped robot; and similarly uses the above-mentioned CPG combined with the reinforcement learning architecture to learn a quadruped robot motion control strategy when the legs are normal and undamaged; constructs a strategy selection state machine to adaptively switch the strategy of the quadruped robot when any leg suddenly fails. Figure 1 As shown, a gait control method for a quadruped robot based on CPG and reinforcement learning proposed by the present invention comprises the following steps:

[0052] S1. Import the overall URDF model of the quadruped robot into the isaac gym simulation platform and set up multiple joint locking situations to simulate actual leg damage, and design a CPG rhythm controller for three legs;

[0053] S2, construct a high-level reinforcement learning strategy, train this strategy so that it can optimize the CPG oscillator parameters in S1 according to the current task goal, and obtain the optimal relative timing relationship between the undamaged legs of the damaged quadruped robot;

[0054] S3, mapping the output signal of CPG to construct the joint space trajectory of the uninjured leg, which will be used as the prior information for the underlying reinforcement learning;

[0055] S4. Build the underlying reinforcement learning strategy to fine-tune the joint angle trajectory output by CPG according to the current robot state to obtain the target joint angle at each moment to ensure that the body balance and terrain traversal ability can be maintained under the premise of current joint damage;

[0056] S5, inputting the final joint angles after fine-tuning into the PD controller to obtain the joint torque, and the joint torque will be used as the gait control signal of the quadruped robot to drive the quadruped robot;

[0057] S6. Construct a strategy selection state machine based on a neural network to adaptively switch the motion strategy of the quadruped robot according to specific situations.

[0058] In this embodiment, the CPG rhythm controller uses a Hopf oscillator, and an oscillator is allocated to each leg of the quadruped robot. The parameters of each oscillator reflect the rhythmic motion information of the leg. At the same time, the phase difference between the undamaged legs is also reflected by two parameters. In the present invention, only the case of single leg damage is considered. Taking the left hind leg damage as an example, the phase difference between the diagonal legs, i.e., the left front leg and the right hind leg, is Δt1, and the phase difference between the two legs on the undamaged side, i.e., the left front leg and the right front leg, is Δt2. The same is true for the remaining legs when they are damaged. The oscillator can generate periodic oscillation signals at the undamaged leg joints, providing a good and feasible quadruped robot motion prior for the underlying reinforcement learning fine-tuning. According to the target CPG parameters given by the high-level reinforcement learning neural network, the rough joint motion trajectory prior is planned, so that the robot can move forward as required; it is necessary to correct the prior trajectory according to the joint angle residual information given by the underlying reinforcement learning neural network, so that the robot motion is more stable and can cross a variety of terrains.

[0059] Due to the development of artificial intelligence, the theories of deep learning and reinforcement learning have begun to be applied to robot kinematic planning. The characteristic of deep reinforcement learning is that it can generate an experience pool by constantly interacting with the environment for network updates. However, when facing some more complex tasks, the requirements for reward design are very high without sufficient prior knowledge. The present invention adopts the SAC algorithm as the reinforcement learning algorithm.

[0060] The CPG network is composed of multiple neurons coupled to each other. There is a stimulating or inhibiting interaction between the neurons in the network, so that each group of signals can produce a stable phase difference. CPG neurons are usually implemented using nonlinear oscillators. The present invention uses a Hopf oscillator to construct a CPG network. Compared with other oscillators, the oscillator model has fewer parameters, its limit cycle is more stable, and it can quickly recover stability when subjected to minor disturbances; the model's frequency, amplitude, support phase and swing phase ratio and other parameters are easy to adjust and analyze. The mathematical model of the oscillator is as follows:

[0061]

[0062] Where, μ (μ>0) is the amplitude of the oscillator output, ω is the frequency of the signal, x and y are the two state variables of the oscillator, ω st With ω sw They represent the frequencies of the swing phase and the support phase respectively, c is a positive number, α represents the convergence rate, the larger its value is, the faster the limit cycle converges.

[0063] The present invention sets an oscillator for each leg, and the coupling between the legs is determined by an additional phase difference parameter. When the legs are not damaged, the quadruped robot usually uses known gaits such as trot, bound, walk, and pace (the phase difference between the legs is known and can be directly set to a constant value), while the present invention focuses on the situation of leg damage (three legs are normal), so the focus of the present invention is to study how to set the oscillator parameters of each undamaged leg and the phase difference between the legs, so that the robot can better complete the task objectives (follow speed instructions, terrain crossing).

[0064] According to the present invention, both high-level and low-level reinforcement learning strategy networks are composed of fully connected neural networks; the input of the high-level reinforcement learning strategy network includes the observation values ​​perceived by the robot body and the external reference speed instructions, and in addition, the environmental privileged information including ground friction and local elevation is also input in the simulation; the input of the low-level reinforcement learning strategy network only includes the observation values ​​perceived by the robot body, including the robot's current speed, posture angle, angles of each joint, angular velocity, and the contact state between the foot end and the ground.

[0065] The hierarchical gait control method of the quadruped robot with impaired legs based on CPG and deep reinforcement learning proposed in this embodiment adopts the PPO reinforcement learning algorithm for both high-level and low-level strategies. The PPO reinforcement learning algorithm model includes: a strategy network and an evaluation network. The input of the high-level reinforcement learning strategy consists of privileged information under the simulator and robot proprioception. The privileged information under the simulator includes the mass of the robot, the ground friction coefficient, the local elevation information of the ground near the robot, etc. The robot proprioception includes the current speed of the robot, the posture angle of the robot, the angle and angular velocity of each joint, and the contact state between the foot end and the ground, etc. In addition, the reference speed instruction and the CPG parameters of the previous step are also part of the high-level reinforcement learning input; the high-level reinforcement learning strategy outputs the current CPG parameters, including the thigh swing amplitude, period, duty cycle, and the relative phase difference of the remaining three legs (expressed by two numerical values). The input of the low-level reinforcement learning strategy includes the current speed of the robot, the posture angle, the angle and angular velocity of each joint, and the contact state of the foot end, and outputs the residual angle of 12 joints, which will be accumulated with the coarse joint space trajectory formed by the high-level CPG parameters to obtain the final joint target trajectory.

[0066] The high-level strategy fully connected neural network is divided into four layers. For the high-level strategy with damaged legs, the number of neurons is 51, 256, 256, and 5 respectively, and the output is the CPG parameters, including the thigh swing amplitude, period, duty cycle, and the relative phase difference of the remaining three legs; for the normal high-level strategy that is not damaged, the number of neurons in the input layer will be added with the joint information of the normal leg and finally become 60 dimensions, and the number of output neurons in the last layer is 4, that is, the two-bit relative phase difference is replaced by a single scalar number of 0 to 3 gait types that can represent trot, bound, walk, and pace.

[0067] The underlying strategy fully connected neural network is also divided into four layers, with 36, 256, 256, and 12 neurons respectively. The output joint angle deviation value is added to the reference trajectory output by the CPG oscillator to obtain the final target joint angle.

[0068] The reward function of the high-level reinforcement learning policy network is used to achieve task rewards such as speed tracking and terrain crossing, while the reward function of the low-level reinforcement learning policy network is used to maintain the balance of the body.

[0069] The reward term used to express speed tracking in the high-level reinforcement learning strategy reward function is:

[0070]

[0071] Where: Indicates the speed command in the x and y directions; v xy Indicates the current speed of the robot in the x and y directions; σ is a shaping scalar, which is 0.25 in this embodiment;

[0072] The reward term used to express terrain traversal in the high-level reinforcement learning policy reward function is:

[0073] r2=n success

[0074] Where: n success Indicates the number of agents that successfully traverse the terrain in the simulation.

[0075] In addition, regularized reward items such as joint acceleration limit, action change rate, and smoothness need to be added to the reward function, as follows:

[0076]

[0077] In the formula, Represents the joint angular acceleration.

[0078]

[0079] In the formula, a t represents the motor joint action at the current time step, at-1 Represents the motor joint action at the previous time step.

[0080]

[0081] In the formula, a t-2 represents the motor joint action two time steps ago.

[0082] The reward term used to represent the balance of the organism in the underlying reinforcement learning strategy reward function is:

[0083]

[0084] Where: v z Indicates the robot's speed in the z direction; ω xy Indicates the angular velocity of the robot in the row and pitch directions.

[0085] Similarly, the underlying reinforcement learning also needs to add the same regularization terms as the previous high-level ones to ensure that the robot's movements are natural enough.

[0086] During the reinforcement learning training process, the robot adopts the form of course training. The courses are set from low difficulty to high difficulty in various scenarios including steps, slopes, unstructured roads, etc., and the training is alternating to enhance the robot's stability and ability to adapt to the terrain.

[0087] In order to successfully transfer the policies trained in the simulator to the real world, we use the teacher-student policy distillation framework. We distill the high-level and low-level policy networks obtained above, use the historical observations and action sequences of the past 50 frames to replace the prior observations such as friction coefficient and elevation value in the simulator, and obtain a hierarchical learning network that can be deployed in the real world through supervised learning.

[0088] When the quadruped robot is moving normally, the normal quadruped robot motion control strategy is loaded, and the quadruped robot motor angle is detected in real time to see if there is any abnormality. When a motor fault is detected, it will automatically switch to the motion control strategy of the quadruped robot with damaged legs to achieve autonomous switching of strategies.

[0089] The present invention constructs a CPG rhythm controller for a damaged model; trains a high-level reinforcement learning model of a quadruped robot to optimize parameters of a CPG oscillator; maps and processes the signal output by the CPG to construct a trajectory of joint angle changes of each undamaged leg; trains an underlying reinforcement learning model of the quadruped robot, fine-tunes the joint angle trajectory output by the CPG according to the current robot state, and inputs the fine-tuned joint angle into a PD controller to obtain joint torque, which is used as a gait control signal of the quadruped robot to drive the quadruped robot; uses the above-mentioned CPG combined with a reinforcement learning architecture to learn a quadruped robot motion control strategy when a normal leg is not damaged; and constructs a strategy selection state machine to adaptively switch the strategy of the quadruped robot when any leg suddenly fails.

[0090] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A motion control method for a quadruped robot with damaged legs, characterized in that: The method constructs a CPG rhythm controller based on a quadruped robot model; When any leg of the quadruped robot fails, the normal quadruped robot motion strategy of the quadruped robot and the leg-damaged quadruped robot motion strategy are switched to control the motion of the quadruped robot; The motion strategies of the normal quadruped robot and the quadruped robot with damaged legs adopt a two-layer reinforcement learning framework: High-level reinforcement learning generates CPG parameters based on the current robot state and reference commands, and the CPG parameters generate joint reference trajectories; The underlying reinforcement learning fine-tunes the reference joint trajectory generated by CPG to obtain the target joint angle at each moment; The joint reference trajectory and the target joint angle are accumulated to obtain the current robot joint torque command.

2. The motion control method for a quadruped robot with damaged legs according to claim 1, characterized in that: The CPG rhythm controller of the quadruped robot adopts a Hopf oscillator. An oscillator is assigned to each leg of the quadruped robot. The parameters of each oscillator reflect the rhythmic motion information of the corresponding leg. The oscillator generates a periodic oscillation signal at the leg joint and obtains the phase difference between the undamaged diagonal legs and between the two legs on the undamaged side when one leg is damaged, providing the quadruped robot motion prior for the underlying reinforcement learning fine-tuning.

3. The motion control method for a quadruped robot with damaged legs according to claim 1, characterized in that: In the two-layer reinforcement learning framework, both the high-level and low-level strategies use the PPO reinforcement learning algorithm: The high-level reinforcement learning policy network takes as input privileged information under the simulator, robot proprioceptive observations, external reference speed instructions, and CPG parameters of the previous step, and outputs target CPG parameters. The target CPG parameters output by high-level reinforcement learning are mapped to construct the coarse joint space trajectory of each undamaged leg; The underlying reinforcement learning policy network inputs the observation values ​​of the robot's proprioception, including the robot's current speed, posture angle, angles and angular velocities of each joint, and the contact status between the foot and the ground; Output the residual angle of each joint; The residual angle output by the bottom-level reinforcement learning is added to the coarse joint space trajectory formed by the target CPG parameters output by the high-level reinforcement learning to obtain the final joint target trajectory.

4. The motion control method for a quadruped robot with damaged legs according to claim 3, characterized in that: In the input of the high-level reinforcement learning strategy network, the privileged information under the simulator includes the current speed of the robot, the mass of the robot, the ground friction coefficient, and the local elevation information of the ground near the robot; the robot proprioception includes the current speed of the robot, the robot posture angle, the angle and angular velocity of each joint, and the contact state between the foot end and the ground; The high-level reinforcement learning strategy network output target CPG parameters of the normal quadruped robot motion strategy include: thigh swing amplitude, period, duty cycle, and gait type; The high-level reinforcement learning strategy network outputs the target CPG parameters of the leg-impaired quadruped robot's motion strategy, including thigh swing amplitude, period, duty cycle, and the relative phase difference of the remaining three legs.

5. The motion control method for a quadruped robot with damaged legs according to claim 3, characterized in that: The reward function of the high-level reinforcement learning policy network is used to implement the task rewards including speed tracking and terrain traversal: The bonus items for speed tracking are: Where: Indicates the speed command in the x and y directions; v xy represents the current velocity of the robot in the x and y directions; σ is a shaping scalar; The rewards for terrain traversal are: r2=n success Where n success Indicates the number of agents that successfully traverse the terrain in the simulation; The reward function of the underlying reinforcement learning policy network is used to maintain the body balance: Where: v z Indicates the robot's speed in the z direction; ω xy Indicates the angular velocity of the robot in the rolling and pitching directions.

6. The motion control method for a quadruped robot with damaged legs according to claim 5, characterized in that: The reward functions of high-level reinforcement learning strategies and low-level reinforcement learning strategies also include regularized reward items including joint acceleration limits, action change rates, and smoothness, as follows: In the formula, represents the joint angular acceleration; In the formula, a t represents the motor joint action at the current time step, a t-1 Represents the motor joint action of the previous time step; In the formula, a t-2 represents the motor joint action two time steps ago.

7. The motion control method for a quadruped robot with damaged legs according to claim 3, characterized in that: The two-layer reinforcement learning framework adopts a teacher-student strategy distillation framework to distill the high-level reinforcement learning model and the strategy network of the high-level reinforcement learning model; uses historical observations and action sequences of past setting frames to replace prior observations in the simulator; and obtains a hierarchical learning network deployed in the real world through supervised learning.

8. The motion control method for a quadruped robot with damaged legs according to claim 1, characterized in that: During the model training process of the dual-layer reinforcement learning framework, a course training form is adopted, and the robot is alternately trained in multiple scenarios to enhance the robot's stability and adaptability to terrain. The scenarios include steps, slopes, and unstructured roads.

9. The motion control method for a quadruped robot with damaged legs according to claim 1, characterized in that: The method step inputs each joint angle in the final joint target trajectory into a PD controller to obtain a joint torque, and uses the joint torque as a quadruped robot gait control signal to drive the quadruped robot.

10. The motion control method for a quadruped robot with damaged legs according to claim 1, characterized in that: The method constructs a strategy selection state machine based on a neural network, and the strategy selection state machine adaptively switches the motion strategy of the quadruped robot according to specific situations.

Citation Information

Cited By

  • Quadruped robot reinforcement learning motion planning method and system based on depth vision

    CN119974024A

  • Method and device for performing dynamic object operation on quadruped robot

    CN120056132A

  • Motion control method for three-foot walking gait of quadruped robot

    CN120363183A

  • A quadruped robot three-legged walking gait motion control method

    CN120363183B

  • Robot control method based on Model-Based and RL

    CN120839805A