Quadruped robot fault-tolerant control method based on explicit predictive feedback

Abstract

The invention provides a quadruped robot fault-tolerant control method based on explicit predictive feedback, and aims to solve the problem that a transferable robot fault-tolerant control model cannot be obtained due to the fact that an explainable and transferable representation containing robot motion model information cannot be obtained in an existing method. The state of the next moment is explicitly predicted by using historical observation information and the current action, and a prediction error is obtained by using an observation true value at the next moment. Prediction errors containing robot motion model information are introduced into input of a motion strategy, a feedback structure is formed, the perceptual ability for the motion model and the environment is provided for a motion network, and a brand-new high-performance fault-tolerant control model which does not need explicit damage judgment and has mobility and interpretability is obtained.

Description

Technical Field

[0001] The present invention relates to the field of quadruped robot motion control based on reinforcement learning, and in particular to a quadruped robot fault-tolerant control method based on explicit predictive feedback. Background Art

[0002] With the continuous improvement of the hardware capabilities and intelligence level of quadruped robots, quadruped robots have shown high flexibility and versatility in performing various tasks in complex environments, and have broad application scenarios in real-life mission scenarios such as search and rescue and patrol. However, quadruped robots are often deployed in unstructured complex environments, so accidents may occur at any time, resulting in damage to the key hardware of quadruped robots. These failures may cause the quadruped robot to lose its movement function and be unable to complete the task.

[0003] When faced with the fault-tolerant control problem of quadruped robots under hardware failures, especially the fault-tolerant control problem under joint failures, traditional control methods need to perform kinematic modeling of the robot under a variety of possible faulty joints and fault conditions, and design or solve the motion control strategy for the corresponding conditions in advance through optimization methods. In the actual deployment process of traditional control methods, quadruped robots need to perform fault self-detection during operation. When a joint failure is detected, the fault characteristics are extracted through sensor data, the faulty joint and fault condition are determined, and then the quadruped robot is switched to the corresponding fault-tolerant control strategy. Such a method is not only complex in structure and cumbersome in process, but also has poor actual deployment effect.

[0004] As a new data-driven control method, reinforcement learning uses neural networks to model and train the controlled object and controller, eliminating the complex modeling and design work of the controlled object and controller, and can obtain a closed-loop neural network controller with strong adaptability and high stability. It is a popular method in the field of legged robot motion control. In "Saving the Limping: Fault-tolerant Quadruped Locomotion via Reinforcement Learning" and "Learning Quadrupedal Locomotion with Impaired Joints Using Random Joint Masking", Liu et al. and Kim et al. used an autoencoder to automatically encode external environmental information and joint fault information based on the teacher-student dual-stage reinforcement learning training paradigm, so that the control strategy gets rid of the fault feature recognition part in the traditional method and realizes the automatic generation of fault-tolerant control strategy. However, in the teacher-student structure training paradigm, a latent vector generated by self-encoding is used to reflect environmental changes and changes in the robot motion model. Such a representation makes the teacher-student structure training paradigm not transferable in different training networks, and also lacks interpretability.

[0005] Outside the research field, many people have recently realized fault-tolerant control of quadruped robots under joint failures based on reinforcement learning. Ai Qinglin et al. (publication number: CN117518821A) predicted joint values ​​through the motor model network, and compared them with the actual robot joint torque values ​​to extract joint motor fault characteristics; designed the quadruped robot fault mechanism according to the fault type, and trained it through the Trust Region Policy Optimization (TRPO) reinforcement learning algorithm. Zhang Weinan et al. (publication number: CN117518821A) proposed a two-stage algorithm, which uses prior information to optimize the exploration space in the exploration learning stage to obtain the optimal solution for each sub-case, and then integrates the robustness operations of multiple fault conditions into a single model in the strategy integration stage. The above methods either need to collect data on real robots for training, lack the generalization of strategies to directly transfer between different robots; or need to use a multi-stage training paradigm, which makes the data utilization rate in the training stage low.

[0006] In addition, in Multi-Task Learning of Active Fault-Tolerant Controller for Leg Failures in Quadruped Robots, Hou et al. used a hierarchical reinforcement learning structure and a state discriminator to select the most suitable strategy in real time among the three motion strategies of normal joints, joint failure, and joint locking; in Meta-Learning for Fast Adaptive Locomotion with Uncertainties in Environments and Robot Dynamics, Anne et al. used meta-reinforcement learning to train a set of original strategy parameters that can quickly iterate to motion strategies under multiple joint failures, which can achieve real-time learning of motion strategies and adapt to joint damage during deployment. However, these methods have not been able to completely solve the above problems.

[0007] In summary, there are various fault-tolerant control methods for joint faults in quadruped robots, many of which have achieved good results, and many of which are based on reinforcement learning. However, none of these methods can achieve an end-to-end single-stage training fault-tolerant control strategy. Quadruped robots still need a dynamic adaptive fault-tolerant control method that is transferable and explainable. Summary of the invention

[0008] Aiming at the problem that existing methods cannot obtain an interpretable and transferable representation containing robot motion model information, and thus cannot obtain a transferable robot fault-tolerant control model; the present invention proposes a quadruped robot fault-tolerant control method based on explicit predictive feedback, especially a quadruped robot fault-tolerant control algorithm for joint failure.

[0009] The present invention is achieved through the following technical solutions:

[0010] A fault-tolerant control method for a quadruped robot based on explicit predictive feedback comprises the following steps:

[0011] S1. Build a reinforcement learning simulation training environment, build a training terrain in the simulation training environment, and design a reward function that meets the single joint failure task;

[0012] S2. Design a single-joint zero-torque loss training course, add a speed tracking term to the reward function, dynamically simulate joint failure by introducing a torque mask matrix and performing dot multiplication with the original torque matrix output by the PD controller; generate a random identification matrix using random integer vectors and one-hot encoding to randomly select faulty joints during training; dynamically adjust the torque mask matrix according to the feedback of the speed tracking reward in each training iteration;

[0013] S3. Build a motion network structure based on explicit prediction feedback, and by expanding the input dimension of the motion network, the quadruped robot can improve its perception of motion model changes from input data;

[0014] S4. Using the simulation training environment built in step S1 and the training course designed in step S2, the motion network expanded in step S3 is trained based on reinforcement learning, and the trained motion strategy is deployed on the quadruped robot.

[0015] Furthermore, the step S1 comprises:

[0016] S11. Build a reinforcement learning simulation training environment, build a training terrain in the simulation training environment, and design a reward function that meets the single joint failure task; select reinforcement learning training algorithms and parameters;

[0017] S12, conducting training tests for normal gait, observing whether the configured training environment, training algorithm and parameters can normally train a movement strategy with normal movement ability, and adjusting the algorithm and parameters according to the training performance;

[0018] S13. Lose any joint torque and conduct training again to observe whether the set reward function can train a usable fault motion strategy, and modify the reward function setting according to the performance, so that the configured training environment, training algorithm and parameters have the ability to train a fault motion strategy.

[0019] Furthermore, the step S2 comprises:

[0020] S21, adding a target speed instruction to the simulation training environment, and adding a speed tracking reward item to the reward function composition, and using the speed tracking reward as a basis for judging the robot's motion ability;

[0021] S22. In the training preparation stage, a random identification matrix is ​​initialized to identify the joint serial numbers corresponding to the damaged joints in each environment during training;

[0022] S23, initializing a torque mask matrix for recording whether each joint currently needs to lose torque and the ratio of the torque that needs to be lost;

[0023] S24. During the training process, after each trajectory sampling, the speed tracking item reward is calculated; if the average reward of the speed tracking item reaches more than 80% of the maximum reward, it is considered that the strategy has the ability to walk in a normal gait, and the torque mask matrix in step S23 is changed according to the random identification matrix recorded in step S22, so that the environment that achieves the goal enters the training of the faulty gait, while ensuring that at most one random joint loses torque in an environment.

[0024] Furthermore, in step S3, the explicit prediction feedback network structure predicts the future state by establishing an explicit estimation network, thereby extracting the information of the motion model; the error between the predicted state and the actual state is provided to the action network as an additional input to form a feedback structure, and the information related to the motion model is added to the action network, thereby improving the capability of the motion network.

[0025] Furthermore, the step S3 comprises:

[0026] S31, establish a historical observation cache to cache the state of the recent period as a source for extracting information related to the robot motion model;

[0027] S32, taking the observation data of the last few frames and the current action as input, obtaining the explicit prediction value of the state at the next moment through an explicit estimation network, thereby obtaining a representation containing the motion model information;

[0028] S33, at the next moment, the prediction error is obtained by subtracting the prediction observation at the previous moment from the current observation, and then the prediction error and the historical observation are sent to the action network together to obtain the action output at the current moment;

[0029] S34. Use the loss of the reinforcement learning algorithm to train the action network, and use the prediction error to train the explicit estimation network.

[0030] Compared with the prior art, the method of the present invention has the following beneficial effects: (1) The joint fault-tolerant control method of the present invention based on explicit predictive feedback can add single-joint fault motion skills on the basis of any original motion model training environment without establishing a separate fault-tolerant control motion model; (2) The training process of the method has only one stage, which greatly improves the data utilization rate in the training stage and ensures that the motion ability of the control model is not weakened due to the knowledge distillation operation; (3) When obtaining the real-time information of the robot motion model, the method uses a neural network to establish an environmental model in the Markov Decision Process (MDP), so that the prediction error can be used to contain the change information of the robot's real-time motion model, which has higher interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0032] Figure 2 It is the principle diagram of executing the fault course described in step 2 of the present invention;

[0033] Figure 3 It is the explicit prediction feedback network structure diagram described in step three of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below in conjunction with specific embodiments and illustrations. The following content will help those skilled in the art to further understand the present invention, but does not limit the present invention in any form.

[0035] The present invention provides a quadruped robot fault-tolerant control method based on explicit predictive feedback, such as Figure 1 As shown, the specific implementation details of this method are as follows:

[0036] Step 1: Build a reinforcement learning simulation training environment, build a training terrain in the simulation training environment, and design a reward function that meets the single-joint failure task.

[0037] S11, use action dimension size n action =12 as the training robot, and build a reinforcement learning simulation training environment based on proximal policy optimization (PPO), in which the action network (Actor) and the critic network (Critic) are composed of multilayer perceptron (MLP). Build and use the URDF file of Jueying lite3 as the training model, and build rugged, staircase, discrete and other terrains in the simulation training environment, add friction coefficient, external force disturbance, center of gravity offset and other domain randomization items to increase the robustness of the final trained strategy. And design the reward function according to the task setting of quadruped robot reinforcement learning, and select the reinforcement learning training algorithm and parameters;

[0038] S12. Use the reinforcement learning simulation training environment built in step S11 to perform training tests on the trot gait (normal gait) of the quadruped robot. Observe the training performance of the configured training environment, training algorithm and algorithm parameters, adjust parameters such as algorithm and strategy entropy, reward function coefficient, and obtain a basic training algorithm that can be trained quickly and does not rely on random number seeds;

[0039] S13. Modify the program of the simulation training environment in step S12, fix the torque of a fixed joint to 0 in the torque calculation link, and modify the reward function calculation at the same time, wherein the reward function calculated based on the joint data needs to exclude the damaged joint for calculation.

[0040] After the modification is completed, training is performed again to observe whether the configurations such as parameters and reward functions can train a usable fault-tolerant control motion strategy, and further modify the parameter settings based on the training performance so that the configured training environment, training algorithm and parameters have the ability to train a fault-tolerant motion strategy.

[0041] Step 2: Design a single-joint zero-torque loss training course, add a speed tracking term to the reward function, and dynamically simulate joint failure by introducing a torque mask matrix and performing dot multiplication with the original torque matrix output by the PD controller; use random integer vectors and one-hot encoding to generate a random identification matrix for randomly selecting faulty joints during training; in each training iteration, dynamically adjust the torque mask matrix based on the feedback of the speed tracking reward.

[0042] In step 2, a joint failure course is added to this embodiment. The working principle of the entire joint failure course is as follows: Figure 2 As shown in the figure, during training, the original moment matrix τ and the moment mask matrix M calculated by the proportional derivative (PD) controller mask Perform point multiplication and then obtain the resulting moment matrix τ masked Input into the simulation training environment built in step 1 for execution. In this operation logic, this embodiment changes M mask Implement joint failure courses.

[0043] S21. If there is no speed tracking reward in the reward function, add a target speed instruction v to the simulation training environment targ , and add the speed tracking reward term R in the reward function v , the speed tracking reward is used as the basis for judging the robot's movement ability;

[0044]

[0045] Where k, σ are the speed tracking reward parameters; v is the current speed;

[0046] S22, in the training preparation stage, first obtain a length of the environment number n env A vector of random integers Each element d i ∈[0,n action -1]. Using one-hot encoding on this random integer vector, we can obtain a random identity matrix M that remains constant throughout the training process. sign , used to identify the joint number corresponding to the joint to be damaged when a failure occurs on a joint of each quadruped robot;

[0047] S23. Initialize a moment mask matrix M mask For M sign The all-one matrix of the same shape is used to record whether each joint needs to fail (i.e., lose torque) and the proportion of torque that needs to be lost;

[0048] S24: During the training process, after each sampling of the trajectory used for training, calculate the speed tracking reward R defined in step S21 v .like Then modify the moment mask matrix M mask , so that M mask The i-th ambient mask vector in minus the random identity matrix M sign The corresponding slice value in M ​​is limited once to avoid mask The value that is not in the range of [0,1] appears in . This makes the environment that achieves the goal enter the training of the fault gait, while ensuring that an environment has at most one random joint loss torque.

[0049] Step 3: Build a motion network structure based on explicit prediction feedback. By expanding the input dimension of the motion network, the quadruped robot can improve its ability to perceive changes in the motion model from the input data.

[0050] In step three, an explicit prediction feedback network structure is added to this embodiment. The explicit prediction feedback network structure predicts the future state by establishing an explicit estimation network, thereby extracting information about the motion model; the error between the predicted state and the actual state is provided as an additional input to the action network to form a feedback structure, which adds information related to the motion model to the action network and improves the motion network capability. Figure 3 shown.

[0051] S31. Establish a historical observation cache to cache the observations of the last 5 frames. t ,s t-1 ,…,s t-4 , as a source for extracting information related to the robot motion model;

[0052] S32, at time t, take the observation s of the last 5 frames t ,s t-1 ,…,s t-4 and the current action a t As input, the explicit prediction value of the next state is obtained through the explicit estimation network built by MLP Thus, a representation containing information about the motion model is obtained;

[0053] S33, at time t+1, use the current observation s t+1 Subtract the current observation prediction with the gradient separated Get the prediction error Then the prediction error δs t+1 With historical observations t+1 ,s t ,…,s t-3 They are sent together to the action network with expanded input dimension to obtain the action output a at time t+1t+1 ;

[0054] S34. Use the loss of the Proximal Policy Optimization (PPO) reinforcement learning algorithm to train the action network, and use the prediction error δs to train the explicit estimation network.

[0055] Step 4: Using the simulation training environment built in step 1 and the training course designed in step 2, use the PPO algorithm to perform reinforcement learning training on the motion network and explicit estimation network in step 3, and transfer the trained motion strategy to the physical robot for deployment and testing.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0057] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A fault-tolerant control method for a quadruped robot based on explicit predictive feedback, characterized in that: The following steps are involved: S1. Build a reinforcement learning simulation training environment, build a training terrain in the simulation training environment, and design a reward function that meets the single joint failure task; S2. Design a single-joint zero-torque loss training course, add a velocity tracking term to the reward function, and dynamically simulate joint failure by introducing a torque mask matrix and performing dot multiplication with the original torque matrix output by the PD controller; Generate a random identification matrix using random integer vectors and one-hot encoding to randomly select faulty joints during training; In each training iteration, the moment mask matrix is ​​dynamically adjusted based on the feedback of the velocity tracking reward; S3. Build a motion network structure based on explicit prediction feedback, and by expanding the input dimension of the motion network, the quadruped robot can improve its perception of motion model changes from input data; S4. Using the simulation training environment built in step S1 and the training course designed in step S2, the motion network expanded in step S3 is trained based on reinforcement learning, and the trained motion strategy is deployed on the quadruped robot.

2. The fault-tolerant control method of a quadruped robot based on explicit predictive feedback according to claim 1, characterized in that: The step S1 comprises: S11. Build a reinforcement learning simulation training environment, build a training terrain in the simulation training environment, and design a reward function that meets the single joint failure task; select reinforcement learning training algorithms and parameters; S12, conducting training tests for normal gait, observing whether the configured training environment, training algorithm and parameters can normally train a movement strategy with normal movement ability, and adjusting the algorithm and parameters according to the training performance; S13. Lose any joint torque and conduct training again to observe whether the set reward function can train a usable fault motion strategy, and modify the reward function setting according to the performance, so that the configured training environment, training algorithm and parameters have the ability to train a fault motion strategy.

3. The fault-tolerant control method of a quadruped robot based on explicit predictive feedback as claimed in claim 1, characterized in that: The step S2 comprises: S21, adding a target speed instruction to the simulation training environment, and adding a speed tracking reward item to the reward function composition, and using the speed tracking reward as a basis for judging the robot's motion ability; S22. In the training preparation stage, a random identification matrix is ​​initialized to identify the joint serial numbers corresponding to the damaged joints in each environment during training; S23, initializing a torque mask matrix for recording whether each joint currently needs to lose torque and the ratio of the torque that needs to be lost; S24. During the training process, after each trajectory sampling, the speed tracking item reward is calculated; if the average reward of the speed tracking item reaches more than 80% of the maximum reward, it is considered that the strategy has the ability to walk in a normal gait, and the torque mask matrix in step S23 is changed according to the random identification matrix recorded in step S22, so that the environment that achieves the goal enters the training of the faulty gait, while ensuring that at most one random joint loses torque in an environment.

4. The fault-tolerant control method of a quadruped robot based on explicit predictive feedback according to claim 1, characterized in that: In step S3, the explicit prediction feedback network structure predicts the future state by establishing an explicit estimation network, thereby extracting the information of the motion model; the error between the predicted state and the actual state is provided as an additional input to the action network to form a feedback structure, and the information related to the motion model is added to the action network, thereby improving the ability of the motion network.

5. The fault-tolerant control method of a quadruped robot based on explicit predictive feedback as claimed in claim 4, characterized in that: The step S3 comprises: S31, establish a historical observation cache to cache the state of the recent period as a source for extracting information related to the robot motion model; S32, taking the observation data of the last few frames and the current action as input, obtaining the explicit prediction value of the state at the next moment through an explicit estimation network, thereby obtaining a representation containing the motion model information; S33, at the next moment, the prediction error is obtained by subtracting the prediction observation at the previous moment from the current observation, and then the prediction error and the historical observation are sent to the action network together to obtain the action output at the current moment; S34. Use the loss of the reinforcement learning algorithm to train the action network, and use the prediction error to train the explicit estimation network.

Images