A fault-tolerant control method for quadruped robots based on explicit predictive feedback

Abstract

To address the problem that existing methods are unable to obtain an interpretable and transferable representation that embodies the robot's motion model, and thus a transferable fault-tolerant control model for the robot, this paper proposes a fault-tolerant control method for a quadruped robot based on explicit predictive feedback. This method uses historical observations and the current action to explicitly predict the next state, and then uses the true observation value to obtain the prediction error at the next moment. By introducing the prediction error that embodies the robot's motion model information into the input of the motion strategy, a feedback structure is formed, which provides the motion network with the ability to perceive the motion model and the environment. This results in a new, transferable, interpretable, and highly expressive fault-tolerant control model that does not require explicit damage judgment.

Description

Technical Field

[0001] The present invention relates to the field of quadruped robot motion control based on reinforcement learning, and in particular to a quadruped robot fault-tolerant control method based on explicit predictive feedback. Background Art

[0002] With the continuous improvement of quadruped robots' hardware capabilities and intelligence, they have demonstrated high flexibility and versatility in performing various tasks in complex environments, and have broad application scenarios in real-world missions such as search and rescue and patrol. However, quadruped robots are often deployed in unstructured and complex environments, where accidents can occur at any time, causing damage to the robot's critical hardware. These failures can cause the robot to lose its motion function, making it unable to complete its mission.

[0003] Traditional control methods, when faced with the problem of fault-tolerant control of quadruped robots under hardware failures, especially joint failures, require kinematic modeling of the robot for a variety of possible faulty joints and fault scenarios, and the design or pre-calculation of motion control strategies for the corresponding scenarios through optimization methods. In actual deployment, traditional control methods require quadruped robots to perform self-fault detection throughout operation. When a joint failure is detected, fault characteristics are extracted from sensor data, the faulty joint and fault scenario are determined, and the quadruped robot is then switched to the corresponding fault-tolerant control strategy. This approach is not only complex in structure and cumbersome in process, but also has poor practical deployment results.

[0004] Reinforcement learning, a new data-driven control method, uses neural networks to model and train the controlled object and controller, eliminating the complex modeling and design of the controlled object and controller. This approach can produce highly adaptable and stable closed-loop neural network controllers, making it a popular approach in the field of legged robot motion control. In "Saving the Limping: Fault-tolerant Quadruped Locomotion via Reinforcement Learning" and "Learning Quadrupedal Locomotion with Impaired Joints Using Random Joint Masking," Liu et al. and Kim et al. employed a teacher-student dual-stage reinforcement learning training paradigm, using autoencoders to automatically encode both external environmental information and joint fault information. This eliminates the fault feature recognition required in traditional methods and enables the automatic generation of fault-tolerant control strategies. However, in this teacher-student training paradigm, environmental changes and changes in the robot's motion model are reflected through a latent vector generated by the autoencoder. This representation makes the teacher-student training paradigm non-transferable across different training networks and lacks interpretability.

[0005] Outside of the research field, many have recently used reinforcement learning to achieve fault-tolerant control of quadruped robots under joint failure. Ai Qinglin et al. (Publication No.: CN117518821A) predicted joint values ​​using a motor model network and compared them with the actual robot joint torque values ​​to extract joint motor fault characteristics. They designed a quadruped robot failure mechanism based on the fault type and trained it using the Trust Region Policy Optimization (TRPO) reinforcement learning algorithm. Zhang Weinan et al. (Publication No.: CN117518821A) proposed a two-stage algorithm. In the exploration learning phase, they optimized the exploration space using prior information to obtain the optimal solution for each sub-case. Then, in the policy integration phase, they integrated robust operations for multiple fault conditions into a single model. These methods either require data collection on real robots for training, lacking the generalizability of direct policy transfer between different robots, or require a multi-stage training paradigm, resulting in low data utilization during the training phase.

[0006] Furthermore, in "Multi-Task Learning of Active Fault-Tolerant Controller for Leg Failures in Quadruped Robots," Hou et al. employed a hierarchical reinforcement learning architecture and a state discriminator to select the most appropriate strategy in real time among three motion strategies: normal joints, joint failures, and joint locks. In "Meta-Learning for Fast Adaptive Locomotion with Uncertainties in Environments and Robot Dynamics," Anne et al. employed meta-reinforcement learning to train a set of primitive policy parameters that can be quickly iterated to motion strategies under various joint failures. This enabled real-time learning of motion strategies during deployment and adaptation to joint damage. However, these approaches have not completely resolved the aforementioned issues.

[0007] In summary, there are currently a variety of fault-tolerant control methods for quadruped robots' joint faults. Many have achieved good results, and many are based on reinforcement learning. However, none of these methods has achieved an end-to-end, single-stage training fault-tolerant control strategy. Quadruped robots still need a dynamic, adaptive, and interpretable fault-tolerant control method. Summary of the Invention

[0008] Aiming at the problem that existing methods cannot obtain an interpretable and transferable representation containing robot motion model information, and thus cannot obtain a transferable robot fault-tolerant control model; the present invention proposes a quadruped robot fault-tolerant control method based on explicit predictive feedback, especially a quadruped robot fault-tolerant control algorithm for joint failure.

[0009] The present invention is achieved through the following technical solutions:

[0010] A fault-tolerant control method for a quadruped robot based on explicit predictive feedback comprises the following steps:

[0011] S1. Build a reinforcement learning simulation training environment, construct a training terrain in the simulation training environment, and design a reward function that meets the single-joint failure task;

[0012] S2. Design a single-joint zero-torque loss training program. Add a velocity tracking term to the reward function. Dynamically simulate joint failure by introducing a torque mask matrix and performing a dot product with the original torque matrix output by the PD controller. Use a random integer vector and one-hot encoding to generate a random identification matrix, which is used to randomly select faulty joints during training. In each training iteration, dynamically adjust the torque mask matrix based on the feedback of the velocity tracking reward.

[0013] S3. Build a motion network structure based on explicit prediction feedback. By expanding the input dimension of the motion network, the quadruped robot can improve its ability to perceive changes in the motion model from the input data.

[0014] S4. Using the simulation training environment built in step S1 and the training course designed in step S2, the motion network expanded in step S3 is trained based on reinforcement learning, and the trained motion strategy is deployed on the quadruped robot.

[0015] Furthermore, the step S1 includes:

[0016] S11. Build a reinforcement learning simulation training environment, construct a training terrain in the simulation training environment, and design a reward function that meets the single-joint failure task; select the reinforcement learning training algorithm and parameters;

[0017] S12. Conduct training tests for normal gait to observe whether the configured training environment, training algorithm, and parameters can normally train a movement strategy with normal movement ability, and adjust the algorithm and parameters based on the training performance;

[0018] S13. Lose any joint torque and train again to observe whether the set reward function can train a usable fault motion strategy. Modify the reward function setting based on the performance so that the configured training environment, training algorithm and parameters have the ability to train a fault motion strategy.

[0019] Furthermore, the step S2 includes:

[0020] S21. Add a target speed command to the simulation training environment, add a speed tracking reward item to the reward function, and use the speed tracking reward as the basis for judging the robot's motion ability;

[0021] S22. In the training preparation phase, a random identification matrix is ​​initialized to identify the joint serial numbers corresponding to the damaged joints in each environment during training;

[0022] S23, initializing a torque mask matrix for recording whether each joint currently needs to lose torque and the ratio of the torque that needs to be lost;

[0023] S24. During the training process, after each trajectory sampling, the speed tracking reward is calculated; if the average reward of the speed tracking item reaches more than 80% of the maximum reward, it is considered that the strategy has the ability to walk with a normal gait. The torque mask matrix in step S23 is changed according to the random identification matrix recorded in step S22, so that the environment that reaches the target enters the training of the fault gait, while ensuring that at most one random joint loses torque in an environment.

[0024] Furthermore, in step S3, the explicit prediction feedback network structure predicts the future state by establishing an explicit estimation network, thereby extracting information about the motion model; the error between the predicted state and the actual state is provided as an additional input to the action network to form a feedback structure, adding information related to the motion model to the action network, thereby improving the capabilities of the motion network.

[0025] Furthermore, step S3 includes:

[0026] S31. Establish a historical observation cache to cache the state of the most recent period of time as a source for extracting information related to the robot motion model;

[0027] S32, taking the observation data of the last few frames and the current action as input, and obtaining an explicit prediction value of the next moment state through an explicit estimation network, thereby obtaining a representation containing motion model information;

[0028] S33. At the next moment, the prediction error is obtained by subtracting the previous moment prediction observation from the current observation. The prediction error and the historical observation are then fed into the action network to obtain the action output at the current moment.

[0029] S34. Use the loss of the reinforcement learning algorithm to train the action network and use the prediction error to train the explicit estimation network.

[0030] Compared with the prior art, the method of the present invention has the following beneficial effects: (1) The joint fault-tolerant control method of the present invention, based on explicit predictive feedback, can add single-joint fault motion skills on the basis of any original motion model training environment without establishing a separate fault-tolerant control motion model; (2) The training process of the method has only one stage, which greatly improves the data utilization rate in the training stage and ensures that the motion ability of the control model is not weakened due to the knowledge distillation operation; (3) When obtaining the real-time information of the robot motion model, the method uses a neural network to establish an environmental model in the Markov Decision Process (MDP), so that the prediction error can be used to contain the change information of the robot's real-time motion model, which has higher interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0032] Figure 2 It is a schematic diagram of the execution principle of the fault course described in step 2 of the present invention;

[0033] Figure 3 This is the explicit prediction feedback network structure diagram described in step three of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to specific embodiments and illustrations. The following content will help those skilled in the art to further understand the present invention, but does not limit the present invention in any form.

[0035] The present invention provides a fault-tolerant control method for a quadruped robot based on explicit predictive feedback, such as Figure 1 As shown, the specific implementation details of this method are as follows:

[0036] Step 1: Build a reinforcement learning simulation training environment, build a training terrain in the simulation training environment, and design a reward function that meets the single-joint failure task.

[0037] S11, use action dimension size n action =12 was used as the training robot. A reinforcement learning simulation training environment based on Proximal Policy Optimization (PPO) was built, where both the actor and critic networks were composed of multilayer perceptrons (MLPs). The URDF file of the Jueying lite3 was constructed and used as the training model. Rough, staircase, and discrete terrain were constructed in the simulation training environment. Domain randomization terms such as friction coefficient, external force perturbation, and center of gravity offset were added to increase the robustness of the final trained policy. The reward function was designed according to the task settings of the quadruped robot reinforcement learning, and the reinforcement learning training algorithm and parameters were selected.

[0038] S12. Use the reinforcement learning simulation training environment built in step S11 to train and test the quadruped robot trot gait (normal gait). Observe the training performance of the configured training environment, training algorithm, and algorithm parameters, adjust the algorithm and policy entropy, reward function coefficient, and other parameters to obtain a basic training algorithm that can be trained quickly and does not rely on random number seeds;

[0039] S13. Modify the program of the simulation training environment in step S12, fix the torque of a fixed joint to 0 in the torque calculation link, and modify the reward function calculation. The reward function calculated based on joint data needs to exclude the damaged joint for calculation.

[0040] After the modification is completed, training is performed again to observe whether the configurations such as parameters and reward functions can train a usable fault-tolerant control motion strategy. Based on the training performance, the parameter settings are further modified to ensure that the configured training environment, training algorithm, and parameters are capable of training a fault-tolerant motion strategy.

[0041] Step 2: Design a single-joint zero-torque loss training program, add a velocity tracking term to the reward function, and dynamically simulate joint failure by introducing a torque mask matrix and performing dot multiplication with the original torque matrix output by the PD controller. Use random integer vectors and one-hot encoding to generate a random identification matrix, which is used to randomly select faulty joints during training. In each training iteration, dynamically adjust the torque mask matrix based on the feedback of the velocity tracking reward.

[0042] In step 2, a joint failure course is added to this embodiment. The working principle of the entire joint failure course is as follows: Figure 2 As shown in the figure, during training, the original torque matrix τ and the torque mask matrix M calculated by the proportional derivative (PD) controller are mask Perform point multiplication and then obtain the resulting moment matrix τ masked Input into the simulation training environment built in step 1 for execution. In this operation logic, this embodiment changes M mask Implement joint failure courses.

[0043] S21. If there is no speed tracking reward in the reward function, add a target speed instruction v to the simulation training environment targ , and add the speed tracking reward term R in the reward function v ,speed tracking reward is used as the basis for judging the robot's movement ability;

[0044]

[0045] Where k, σ are the speed tracking reward parameters; v is the current speed;

[0046] S22, in the training preparation stage, first obtain a length of the environment number n env A random integer vector Each element d i ∈[0,n action -1]. Using one-hot encoding on this random integer vector, we can obtain a random identity matrix M that remains constant throughout the training process. sign , used to identify the joint number corresponding to the joint to be damaged when a failure occurs on a joint of each quadruped robot;

[0047] S23. Initialize a moment mask matrix M mask For M sign The all-one matrix of the same shape is used to record whether each joint needs to fail (i.e., lose torque) and the ratio of torque that needs to be lost;

[0048] S24. During the training process, after each sampling of the trajectory used for training, calculate the speed tracking reward R defined in step S21 v .like Then modify the moment mask matrix M mask , so that M mask The i-th ambient mask vector minus the random identity matrix M sign The value of the corresponding slice in M ​​is limited once to avoid mask The value that is not in the range of [0,1] appears in . This makes the environment that achieves the goal enter the training of the fault gait, while ensuring that an environment has at most one random joint loss torque.

[0049] Step 3: Build a motion network structure based on explicit predictive feedback. By expanding the input dimension of the motion network, the quadruped robot can improve its ability to perceive changes in the motion model from the input data.

[0050] In step 3, an explicit prediction feedback network structure is added to this embodiment. The explicit prediction feedback network structure predicts the future state by establishing an explicit estimation network, thereby extracting information about the motion model; the error between the predicted state and the actual state is provided as an additional input to the action network to form a feedback structure, adding information related to the motion model to the action network and improving the motion network capability. The network structure used in this embodiment is as follows Figure 3 shown.

[0051] S31. Create a historical observation cache to cache the observations of the last 5 frames. t ,s t-1 ,…,s t-4 , as a source for extracting information related to the robot motion model;

[0052] S32, at time t, take the observation s of the last 5 frames t ,s t-1 ,…,s t-4 and the current action a t As input, the explicit prediction value of the next moment state is obtained through the explicit estimation network built by MLP Thus, a representation containing information about the motion model is obtained;

[0053] S33, at time t+1, using the current observation s t+1 Subtract the current observation prediction with the gradient separated Get the prediction error Then the prediction error δs t+1 and historical observations t+1 ,s t ,…,s t-3 They are fed into the action network with expanded input dimension to obtain the action output a at time t+1t+1 ;

[0054] S34. Use the loss of the Proximal Policy Optimization (PPO) reinforcement learning algorithm to train the action network, and use the prediction error δs to train the explicit estimation network.

[0055] Step 4: Using the simulation training environment built in step 1 and the training course designed in step 2, use the PPO algorithm to perform reinforcement learning training on the motion network and explicit estimation network in step 3, and transfer the trained motion strategy to the physical robot for deployment and testing.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0057] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A fault-tolerant control method for a quadruped robot based on explicit predictive feedback, characterized in that: The following steps are involved: S1. Build a reinforcement learning simulation training environment, construct a training terrain in the simulation training environment, and design a reward function that meets the single-joint failure task; S2. Design a single-joint zero-torque loss training course, add a velocity tracking term to the reward function, and dynamically simulate joint failure by introducing a torque mask matrix and performing dot multiplication with the original torque matrix output by the PD controller; Generate a random identity matrix using random integer vectors and one-hot encoding to randomly select faulty joints during training; In each training iteration, the moment mask matrix is ​​dynamically adjusted based on the feedback of the velocity tracking reward; S3. Build a motion network structure based on explicit prediction feedback. By expanding the input dimension of the motion network, the quadruped robot can improve its ability to perceive changes in the motion model from the input data. S31. Establish a historical observation cache to cache the state of the most recent period of time as a source for extracting information related to the robot motion model; S32, taking the observation data of the last few frames and the current action as input, and obtaining an explicit prediction value of the next moment state through an explicit estimation network, thereby obtaining a representation containing motion model information; S33. At the next moment, the prediction error is obtained by subtracting the previous moment prediction observation from the current observation. The prediction error and the historical observation are then fed into the action network to obtain the action output at the current moment. S34. Use the loss of the reinforcement learning algorithm to train the action network and the prediction error to train the explicit estimation network; S4. Using the simulation training environment built in step S1 and the training course designed in step S2, the motion network expanded in step S3 is trained based on reinforcement learning, and the trained motion strategy is deployed on the quadruped robot.

2. The fault-tolerant control method for a quadruped robot based on explicit predictive feedback according to claim 1, wherein: The step S1 comprises: S11. Build a reinforcement learning simulation training environment, construct a training terrain in the simulation training environment, and design a reward function that meets the single-joint failure task; select the reinforcement learning training algorithm and parameters; S12. Conduct training tests for normal gait to observe whether the configured training environment, training algorithm, and parameters can normally train a movement strategy with normal movement ability, and adjust the algorithm and parameters based on the training performance; S13. Lose any joint torque and train again to observe whether the set reward function can train a usable fault motion strategy. Modify the reward function setting based on the performance so that the configured training environment, training algorithm and parameters have the ability to train a fault motion strategy.

3. The fault-tolerant control method for a quadruped robot based on explicit predictive feedback according to claim 1, wherein: The step S2 comprises: S21. Add a target speed command to the simulation training environment, add a speed tracking reward item to the reward function, and use the speed tracking reward as the basis for judging the robot's motion ability; S22. In the training preparation phase, a random identification matrix is ​​initialized to identify the joint serial numbers corresponding to the damaged joints in each environment during training; S23, initializing a torque mask matrix for recording whether each joint currently needs to lose torque and the ratio of the torque that needs to be lost; S24. During the training process, after each trajectory sampling, the speed tracking reward is calculated; if the average reward of the speed tracking item reaches more than 80% of the maximum reward, it is considered that the strategy has the ability to walk with a normal gait. The torque mask matrix in step S23 is changed according to the random identification matrix recorded in step S22, so that the environment that reaches the target enters the training of the fault gait, while ensuring that at most one random joint loses torque in an environment.

4. The fault-tolerant control method for a quadruped robot based on explicit predictive feedback according to claim 1, wherein: In step S3, the explicit prediction feedback network structure predicts the future state by establishing an explicit estimation network, thereby extracting information about the motion model; the error between the predicted state and the actual state is provided as an additional input to the action network to form a feedback structure, adding information related to the motion model to the action network, thereby improving the motion network capability.

Images