Quadruped robot fault-tolerant control method and system based on residual learning

Abstract

The invention provides a quadruped robot fault-tolerant control method and system based on residual learning, and the method comprises the steps: constructing an ontology mechanism model based on phase information, dividing the phases of a supporting stage and a swinging stage based on a diagonal gait, and designing foot end tracks through combining a Bezier curve and a sine curve; designing a six-dimensional reward function including speed tracking, posture balance, foot movement direction, energy consumption control, body contact constraint and foot end contact excitation; a data-driven model based on a heterogeneous actor-commentator architecture is constructed, an actor network integrates terrain information, ontology sensing data and damage parameter estimation values, a commentator network integrates privilege information for strategy evaluation, and network parameters are optimized based on a near-end strategy optimization algorithm; and on the basis of a residual learning thought, a final motion instruction is generated by coupling the correction output by the data driving model with the ontology mechanism model.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular, to a fault-tolerant control method and system for a quadruped robot based on residual learning. Background Technique

[0002] In recent years, the robot industry has developed rapidly, and quadruped robots have emerged as a research hotspot. Compared with wheeled or tracked robots, quadruped robots provide stable support and traction with independent landing points, showing significant advantages in dealing with unstructured environments. However, in complex and changing actual application scenarios, robots may suffer injuries such as limb fractures due to harsh external environments, resulting in reduced locomotion ability and limited control performance. To address such challenges, by implementing a fault-tolerant control strategy to dynamically adjust the undamaged parts, it can not only compensate for system failures in real time and ensure safe recovery in emergency situations, but also effectively extend the service life of the equipment, improve the overall task completion rate and reliability.

[0003] Traditional fault-tolerant control of quadruped robots usually includes two steps: first is fault diagnosis, and second is to select the most appropriate alternative strategy based on the diagnosis results, and these strategies are often pre-designed manually. However, due to the diversity of robot damage conditions, it is difficult for manually designed control strategies to cover all possible fault modes. In addition, affected by sensor noise during the fault diagnosis process, the diagnosis results may have errors, further reducing the reliability and applicability of traditional methods.

[0004] With the improvement of computing power, deep reinforcement learning has made breakthrough progress in the field of quadruped robot motion control. Through end-to-end learning, the robot can autonomously adapt to complex terrains (such as gravel roads, slopes, stairs, etc.) and achieve dynamically stable gaits. This method does not rely on accurate fault modeling, but directly extracts features from multi-modal sensor data (such as IMU information, joint encoder information, foot-end force sensor information), and combines diverse fault pre-training in the simulation environment and Sim-to-Real transfer technology, enabling the control strategy to have the ability to adapt to unknown faults. Therefore, the deep reinforcement learning method shows great potential in the fault-tolerant motion control of quadruped robots in the case of limb fractures, providing a new technical path for improving the task execution ability of robots in extreme environments.

[0005] The patent application document CN116449727A discloses a fault-tolerant gait planning method for a hexapod robot based on reinforcement learning, including the following steps: building a simulation model of the hexapod robot; establishing a CPG gait generator based on an improved Hopf oscillator; integrating the simulation model, the policy network and the CPG gait generator based on the improved Hopf oscillator to construct a reinforcement learning motion control framework, and for the established simulation model of the hexapod robot, integrating the reinforcement learning motion control framework; simulating that some legs of the hexapod robot fail, freezing the failed legs of the hexapod robot in the simulation environment, and training the reinforcement learning motion control framework; integrating the trained control network into the gait control framework of the hexapod robot to generate a fault-tolerant gait of the hexapod robot and verify it. However, this patent cannot completely solve the existing technical problems and cannot meet the requirements of the present invention. Summary of the Invention

[0006] Aiming at the defects in the prior art, the purpose of the present invention is to provide a fault-tolerant control method and system for a quadruped robot based on residual learning.

[0007] According to the fault-tolerant control method for a quadruped robot based on residual learning provided by the present invention, it includes:

[0008] Step 1: Construct a quadruped robot body mechanism model based on phase information;

[0009] Step 2: Construct a reward function for fault-tolerant motion control;

[0010] Step 3: Construct a data-driven model based on a heterogeneous actor-critic architecture;

[0011] Step 4: Based on the residual learning framework, train and optimize the data-driven model through the reward function, use the obtained residual information as the residual correction amount of the quadruped robot body mechanism model, and finally generate a motion control instruction for the quadruped robot by coupling the output of the body mechanism model and the residual correction amount.

[0012] Preferably, the step 1 includes:

[0013] The motion cycle of the quadruped robot foot end is defined as T stride , including two stages: the swing stage T sw and the support stage T st , and T stride =T sw +T st ; the duration of the swing stage T sw is manually specified according to specific circumstances, and the time of the support stage is where v d represents a predefined speed, and L span is half of the stride length;

[0014] The core of the body mechanism model lies in determining the phase of each leg. The phase is designed based on the contact information of the reference leg l ref with the ground. When the force of the reference leg in the z-direction exceeds the threshold, it is defined as a touchdown event, represented by the boolean value TD ref to indicate this event, which is true when l ref contacts the ground; during the swing phase, the phase of each leg is calculated using the following formula:

[0015]

[0016] where is the touchdown moment of l ref , represents the time elapsed after

[0017] Adopting a diagonal gait, where the phases of the left front leg and the right hind leg are the same, and the phases of the right front leg and the left hind leg are the same. When choosing the left front leg or the right hind leg as the reference leg, When choosing the right front leg or the left hind leg as the reference leg, as shown in the following formula:

[0018]

[0019] where ΔS ref，i represents the phase difference of leg i relative to l ref ; respectively represent the phase differences of the left front leg FL, the right hind leg HR, the right front leg FR, and the left hind leg HL relative to the reference leg in the diagonal Trot gait; ΔS ref，FL , ΔS ref，FR , ΔS ref，HL , ΔS ref，HR respectively represent the phase differences of the left front leg FL, the right front leg FR, the left hind leg HL, and the right hind leg HR relative to the reference leg;

[0020] Next, normalize the clock signal by mapping the clock t i of each leg i to S i (t). When a certain leg is in the swing phase, 0 ≤ S i (t) ≤ 1, and when in the support phase, 1 ≤ S i (t) ≤ 2. The normalized clock signal function S i (t) is as follows:

[0021]

[0022] where T stIndicates the time period when the foot tip of the quadruped robot is in the support phase; T sw Indicates the time period when the foot tip of the quadruped robot is in the swing phase;

[0023] After determining the phase of each leg, it is used to design the foot tip trajectory of the robot. During the swing phase, a Bezier curve is used as the reference trajectory, and during the support phase, a sine curve is used as the reference trajectory, as follows:

[0024]

[0025] s.t. 0 ≤ S i (t) ≤ 1

[0026]

[0027] s.t. 1 ≤ S i (t) ≤ 2

[0028] Wherein, Is the position of the foot tip of the robot relative to the abduction joint coordinate system during the swing phase, And Respectively represent its coordinates in three directions, Is the k-th control point in the xy plane, Is the k-th control point in the z direction, Represents the Bernstein polynomial of degree n, n is the number of control points, ρ represents the angle of the positive direction of the robot body relative to the target direction, Represents the position of the foot tip during the support phase, σ represents the amplitude of the sine curve; Represents the coordinates of the foot tip in three directions during the support phase;

[0029] By using the inverse kinematics of the robot, the reference position q of the robot joints is determined according to these values ref , as follows:

[0030]

[0031] Wherein, IK(·) represents the inverse kinematic solution of the quadruped robot.

[0032] Preferably, the step 2 includes:

[0033] The reward function contains 6 items, among which the body linear velocity reward item r1 aims to motivate the robot to track the target velocity, and the calculation formula is as follows:

[0034]

[0035] Wherein, Represents the desired linear velocity of the robot body, Represents the actual linear velocity, Represents the desired Euler angles of the body, Φ z Represents the current Euler angles; tanh is the hyperbolic tangent function; k1 is a dynamic weight coefficient based on the attitude error, used to adjust the value of the reward function r1;

[0036] The body attitude reward term r2 imposes a penalty on the rotation of the robot's body in the roll and pitch directions to maintain the balance of the robot's body. Its calculation formula is as follows:

[0037]

[0038] Where, Represents the roll angle Φ of the robot x And the pitch angle Φ y Of the square;

[0039] The foot speed reward term r3 encourages the robot's feet to move in the desired direction. Its calculation formula is as follows:

[0040]

[0041] Where, Represents the target speed of the foot, Represents the actual speed of the foot, w1, w2, and w3 are predefined constants; k3 is a dynamic weight coefficient based on the attitude error, used to adjust the value of the reward function r3;

[0042] The energy loss reward term r4 aims to penalize the motor energy consumption. Its calculation formula is as follows:

[0043]

[0044] Where, τ i Represents the torque of each joint, Represents the speed of each joint, t corresponds to the time required for each time step of the reinforcement learning;

[0045] The body contact reward term r5 is used to impose a penalty on the contact of components other than the feet of the robot with the ground. Its calculation formula is as follows:

[0046]

[0047] Where, Is a boolean value used to indicate whether parts other than the feet of the robot are in contact with the ground; N l Represents the total number of components of the robot;

[0048] The foot tip contact reward term r6 is used to encourage the contact of the robot's feet with the ground. Its calculation formula is as follows:

[0049]

[0050] Among them, is used to indicate whether the foot touches the ground;

[0051] The reward value r for each step of reinforcement learning t is the weighted sum of the above six reward values, as follows:

[0052] r t = 1×r1 + 0.5×r2 + 0.2×r3 - 0.1×r4 - 0.1×r5 - 0.1×r6.

[0053] Preferably, the step 3 includes:

[0054] The data-driven model consists of three parts: an actor network, a critic network, and an identification network; among them, the actor network adopts a multi-layer perceptron architecture, and the number of neurons in each layer of the network is set to [N Actor , 512, 256, 128, 12], where N Actor is the dimension of the input information. The input of this network includes terrain information, body perception information, body structure parameters estimated by the identification network, and body linear velocity information, and the output is the correction amount of the body mechanism model; among them, the body perception information is directly collected by the sensors on the robot, covering the body angular velocity and gravity projection collected by the IMU, the joint angles and joint angular velocities collected by the joint encoders, and the robot motion instruction information sent by the user; the terrain information is obtained by the lidar installed on the robot; design a heterogeneous critic network to improve the training efficiency of the model, and its network architecture is [N Critic , 512, 256, 128, 1]; N Critic represents the dimension of the input information of the Critic network;

[0055] Both the actor network and the critic network adopt the exponential linear unit as the activation function between the hidden layers. The input of the critic network includes privileged information, terrain information, and body perception information, where the privileged information not only includes the real robot body structure parameters and real body linear velocity, but also includes the robot foot end touchdown state, external disturbance force, and the gain parameters of the proportional-derivative controller; the actor network and the critic network are optimized by the proximal policy optimization algorithm, and their loss functions are respectively:

[0056]

[0057] Among them, θ is the parameter of the actor network, and φ is the parameter of the critic network; represents the ratio of the new and old policies; π θ represents the policy being optimized currently, represents the old policy used to collect experience data before; a tDenote the action selected by the agent at time step t; s t Denote the environmental state at time step t; Denote the expectation for the current batch of samples; V φ (s t ) denotes the value estimation of state s by the Critic network t ; A t is the advantage function, calculated by the Generalized Advantage Estimation algorithm; ∈ is the clipping parameter, used to limit the change range of r t ; when A t > 0, it is used to limit the policy improvement amplitude, and when A t < 0, it is used to limit the policy degradation amplitude; V target is the target value function, calculated by the Temporal Difference algorithm;

[0058] Considering that there are differences in the body structures of the robot under different damage conditions, a recognition network is designed to estimate the body structure parameters of the robot in the current damage state in real time and use them as the input of the actor network; the recognition network is constructed using a multi-layer perceptron, and its network architecture is [N identify , 256, 128, 11]; N identify represents the dimension of the input information of the recognition network;

[0059] This network uses the rectified linear unit as the activation function between hidden layers. The input of this network is the historical proprioceptive information, and the output contains two parts: one part is the estimated body structure parameters, and the other part is the estimated body linear velocity; the loss function of this network consists of the regression loss and the loss of the actor network in reinforcement learning These two parts are combined, and the specific form is as follows:

[0060]

[0061] where α is a hyperparameter used to balance different components of the loss function; v t and respectively represent the body linear velocity estimated by the recognition network and the actual body linear velocity; and respectively represent the estimated and actual body structure parameters; MSE() represents the mean squared error loss function;

[0062] The body structure parameters are represented by an 8-dimensional vector, and each value corresponds to the structural states of the left front thigh, left front calf, right front thigh, right front calf, left rear thigh, left rear calf, right rear thigh, and right rear calf respectively. A value of 0 indicates that this part has been completely broken, a value of 1 indicates that this part is intact, and a value between 0 and 1 indicates the proportion of the undamaged part in the whole;

[0063] Based on the idea of residual learning, the joint position residuals output by the data-driven model are coupled and corrected with the reference joint positions generated by the body mechanism model to construct the target joint position commands. To suppress high-frequency interference and improve control continuity, a low-pass Butterworth filter is used to smooth the target joint positions, and then the smooth joint positions are mapped to the target joint torques through a proportional-derivative (PD) tracking controller, and finally the actuator drive signals of the quadruped robot are generated. Among them, the dynamic equation of the PD controller is expressed as:

[0064]

[0065] Among them, and represent the desired joint angles and angular velocities, is set to 0; q j and represent the current joint angle and angular velocity respectively; the gain parameters k p and k d are specified manually.

[0066] Preferably, step 4 includes: after training is completed, the actor network and the identification network are converted from the.pth format to a standard ONNX computation graph through the torch.onnx.export() interface of the PyTorch framework, and the model inference consistency is verified through ONNX Runtime; the ONNX model is parsed using the NvOnnxParser toolchain of TensorRT to construct an inference engine with FP16 precision, and low-latency inference code is written based on the CUDA 11.6 and C++17 standards; to achieve simulation-reality control consistency, the body mechanism model and the data-driven model are integrated and deployed to the NVIDIA Jetson AGX Xavier embedded platform, and a multi-modal input pipeline is constructed through the real-time API interface provided by the quadruped robot manufacturer, and the fused control commands are sent to the underlying motor controller at a frequency of 1 kHz; to eliminate the trajectory jitter caused by discrete control, cubic spline interpolation is used to upsample the 50-Hz policy output to ensure consistency with the simulator step size; finally, the stability of the fault-tolerant strategy is verified in diverse test scenarios, and the algorithm is iteratively optimized according to the test results.

[0067] According to the fault-tolerant control system for quadruped robots based on residual learning provided by the present invention, it includes:

[0068] Module M1: Construct a body mechanism model of the quadruped robot based on phase information;

[0069] Module M2: Construct a reward function for fault-tolerant motion control;

[0070] Module M3: Construct a data-driven model based on a heterogeneous actor-critic architecture;

[0071] Module M4: Based on the residual learning framework, train and optimize the data-driven model through a reward function, use the obtained residual information as the residual correction amount of the quadruped robot's body mechanism model, and finally generate the motion control instructions of the quadruped robot by coupling the output of the body mechanism model and the residual correction amount.

[0072] Preferably, the module M1 includes:

[0073] The motion cycle of the quadruped robot's foot end is defined as T stride , including two phases: the swing phase T sw and the support phase T st , and T stride = T sw + T st ; The duration of the swing phase T sw is manually specified according to specific circumstances, and the time of the support phase is where v d represents the predefined speed, and L span is half of the stride length;

[0074] The core of the body mechanism model is to determine the phase of each leg. The phase is designed based on the contact information of the reference leg l ref with the ground. When the force of the reference leg in the z direction exceeds the threshold, it is defined as a touchdown event, which is represented by the boolean value TD ref to indicate this event, and it is true when l ref contacts the ground; In the swing phase, the phase of each leg is calculated using the following formula:

[0075]

[0076] where, is the touchdown moment of l ref , represents the time elapsed after

[0077] Adopt a diagonal gait, where the phases of the left front leg and the right hind leg are the same, and the phases of the right front leg and the left hind leg are the same. When choosing the left front leg or the right hind leg as the reference leg, When choosing the right front leg or the left hind leg as the reference leg, as shown in the following formula:

[0078]

[0079] where, ΔS ref,i represents the relative displacement of leg i with respect to lref Phase difference; respectively represent the phase differences of the left front leg FL, right hind leg HR, right front leg FR, and left hind leg HL relative to the reference leg in the diagonal trot gait; ΔS ref，FL , ΔS ref，FR , ΔS ref，HL , ΔS ref，HR respectively represent the phase differences of the left front leg FL, right front leg FR, left hind leg HL, and right hind leg HR relative to the reference leg;

[0080] Next, normalize the clock signal by mapping the clock t i of each leg i to S i (t). When a leg is in the swing phase, 0 ≤ S i (t) ≤ 1, and when in the support phase, 1 ≤ S i (t) ≤ 2. The normalized clock signal function S i (t) is as follows:

[0081]

[0082] where, T st represents the time period when the foot end of the quadruped robot is in the support phase; T sw represents the time period when the foot end of the quadruped robot is in the swing phase;

[0083] After determining the phase of each leg, use it to design the foot end trajectory of the robot. In the swing phase, use a Bezier curve as the reference trajectory, and in the support phase, use a sine curve as the reference trajectory, as follows:

[0084]

[0085] s.t. 0 ≤ S i (t) ≤ 1

[0086]

[0087] s.t. 1 ≤ S i (t) ≤ 2

[0088] where, is the position of the foot end of the robot relative to the abduction joint coordinate system in the swing phase, and respectively represent its coordinates in three directions, is the k-th control point in the xy plane, is the k-th control point in the z direction, represents the Bernstein polynomial of degree n, n is the number of control points, and ρ represents the angle of the positive direction of the robot body relative to the target direction, represents the position of the foot end during the support phase, and σ represents the amplitude of the sine curve; represents the coordinates of the foot end in three directions during the support phase;

[0089] By using the inverse kinematics of the robot, the reference position q of the robot joints is determined according to these values ref , as follows:

[0090]

[0091] where IK(·) represents the inverse kinematic solution of the quadruped robot.

[0092] Preferably, the module M2 includes:

[0093] The reward function contains 6 terms. Among them, the body linear velocity reward term r1 aims to encourage the robot to track the target velocity, and its calculation formula is as follows:

[0094]

[0095] where, represents the desired linear velocity of the robot body, represents the actual linear velocity, represents the desired Euler angle of the body, Φ z represents the current Euler angle; tanh is the hyperbolic tangent function; k1 is the dynamic weight coefficient based on the attitude error, used to adjust the value of the reward function r1;

[0096] The body attitude reward term r2 imposes a penalty on the rotation of the robot body in the roll and pitch directions to maintain the balance of the robot body, and its calculation formula is as follows:

[0097]

[0098] where, represents the roll angle Φ of the robot x and the pitch angle Φ y squared;

[0099] The foot velocity reward term r3 encourages the robot's feet to move in the desired direction, and its calculation formula is as follows:

[0100]

[0101] where, represents the target velocity of the foot, represents the actual velocity of the foot, w1, w2 and w3 are predefined constants; k3 is the dynamic weight coefficient based on the attitude error, used to adjust the value of the reward function r3;

[0102] The energy loss reward term r4 is designed to penalize the energy consumption of the motor, and its calculation formula is as follows:

[0103]

[0104] where τ i represents the torque of each joint, represents the speed of each joint, and t corresponds to the time required for each time step of reinforcement learning;

[0105] The body contact reward term r5 is used to impose a penalty on the contact between other components of the robot except the feet and the ground, and its calculation formula is as follows:

[0106]

[0107] where, is a boolean value used to indicate whether other parts of the robot except the feet are in contact with the ground; N l represents the total number of components of the robot;

[0108] The foot end contact reward term r6 is used to encourage the contact between the robot's feet and the ground, and its calculation formula is as follows:

[0109]

[0110] where, is used to indicate whether the feet are in contact with the ground;

[0111] The reward value r for each step of reinforcement learning t is the weighted sum of the above six reward values, as follows:

[0112] r t = 1×r1 + 0.5×r2 + 0.2×r3 - 0.1×r4 - 0.1×r5 - 0.1×r6.

[0113] Preferably, the module M3 includes:

[0114] The data-driven model consists of three parts: an actor network, a critic network, and an identification network; among them, the actor network adopts a multi-layer perceptron architecture, and the number of neurons in each layer of the network is set to [N Actor , 512, 256, 128, 12], where N Actoris the dimension of the input information. The input of this network includes terrain information, proprioceptive information, body structure parameters estimated by the identification network, and body linear velocity information, and the output is the correction amount of the proprioceptive mechanism model. Among them, the proprioceptive information is directly collected by sensors on the robot, covering the body angular velocity and gravity projection collected by the IMU, the joint angles and joint angular velocities collected by the joint encoders, and the robot motion instruction information sent by the user. The terrain information is obtained by the lidar installed on the robot. Design a heterogeneous critic network to improve the training efficiency of the model, and its network architecture is [N Critic , 512, 256, 128, 1]; N Critic represents the dimension of the input information of the Critic network;

[0115] Both the actor network and the critic network use the exponential linear unit as the activation function between hidden layers. The input of the critic network includes privileged information, terrain information, and proprioceptive information. Among them, the privileged information not only includes the real robot body structure parameters and real body linear velocity, but also includes the foot contact state of the robot, the external disturbance force, and the gain parameters of the proportional-derivative controller. The actor network and the critic network are optimized by the proximal policy optimization algorithm, and their loss functions are respectively:

[0116]

[0117] Among them, θ is the parameter of the actor network, and φ is the parameter of the critic network; represents the ratio of the new and old policies; π θ represents the current policy being optimized, represents the old policy used to collect experience data before; a t represents the action selected by the agent at time step t; s t represents the environmental state at time step t; represents the expectation for the current batch of samples; V φ (s t ) represents the value estimation of the state s t by the Critic network; A t is the advantage function, which is calculated by the generalized advantage estimation algorithm; ∈ is the clipping parameter, which is used to limit the change range of r t ; when A t > 0, it is used to limit the policy improvement amplitude, and when A t < 0, it is used to limit the policy degradation amplitude; V target is the target value function, which is calculated by the temporal difference algorithm;

[0118] Considering that the body structure of the robot varies under different damage conditions, a recognition network is designed to estimate the body structure parameters of the robot in the current damage state in real time and use them as the input of the actor network. The recognition network is constructed using a multi-layer perceptron, and its network architecture is [N identify , 256, 128, 11]; N identify represents the dimension of the input information of the recognition network;

[0119] This network uses the rectified linear unit as the activation function between hidden layers. The input of this network is the historical proprioceptive information, and the output contains two parts: one part is the estimated body structure parameters, and the other part is the estimated body linear velocity. The loss function of this network consists of the regression loss and the loss of the actor network in reinforcement learning These two parts are combined, and the specific form is as follows:

[0120]

[0121] where α is a hyperparameter used to balance different components of the loss function; v t and represent the body linear velocity estimated by the recognition network and the actual body linear velocity respectively; and represent the estimated and actual body structure parameters respectively; MSE() represents the mean square error loss function;

[0122] The body structure parameters are represented by an 8-dimensional vector. Each value corresponds to the structural state of the left front thigh, left front calf, right front thigh, right front calf, left rear thigh, left rear calf, right rear thigh, and right rear calf respectively. A value of 0 indicates that this part has been completely broken, a value of 1 indicates that this part is intact, and a value between 0 and 1 indicates the proportion of the undamaged part in the whole;

[0123] Based on the idea of residual learning, the joint position residuals output by the data-driven model are coupled and corrected with the reference joint positions generated by the body mechanism model to construct the target joint position command. To suppress high-frequency interference and improve control continuity, a low-pass Butterworth filter is used to smooth the target joint position, and then the smooth joint position is mapped to the target joint torque through a proportional-derivative (PD) tracking controller, and finally the actuator drive signal of the quadruped robot is generated. Among them, the dynamic equation of the PD controller is expressed as:

[0124]

[0125] where, and represent the desired joint angle and angular velocity, is set to 0; qj and represent the current joint angle and angular velocity respectively; the gain parameters k p and k d are specified manually.

[0126] Preferably, the module M4 includes: after the training is completed, the actor network and the identification network are converted from the.pth format to a standard ONNX computation graph through the torch.onnx.export() interface based on the PyTorch framework, and the model inference consistency is verified through ONNX Runtime; the NvOnnxParser toolchain of TensorRT is used to parse the ONNX model, build an inference engine with FP16 precision, and write low-latency inference code based on CUDA11.6 and C++17 standards; to achieve simulation-reality control consistency, the ontology mechanism model and the data-driven model are integrated and deployed to the NVIDIA Jetson AGX Xavier embedded platform. Through the real-time API interface provided by the quadruped robot manufacturer, a multi-modal input pipeline is constructed, and the fused control instructions are sent to the underlying motor controller at a frequency of 1kHz; to eliminate the trajectory jitter caused by discrete control, cubic spline interpolation is used to upsample the 50Hz policy output to ensure consistency with the simulator step size; finally, the stability of the fault-tolerant strategy is verified in diverse test scenarios, and the algorithm is iteratively optimized according to the test results.

[0127] Compared with the prior art, the present invention has the following beneficial effects:

[0128] (1) By combining the data-driven model with the ontology mechanism model and using residual learning to dynamically correct the output of the mechanism model, it not only retains the advantages of the mechanism model in prior knowledge but also makes up for the problems of sparse positive samples and low training efficiency of the pure data-driven model in fault scenarios, significantly accelerating the policy convergence speed and at the same time improving the control accuracy for specific damage conditions;

[0129] (2) The six designed composite reward functions (speed tracking, attitude balance, foot movement direction, energy consumption suppression, non-foot contact penalty, foot tip touchdown excitation) can comprehensively cover the physical constraints and goals of fault-tolerant control, and guide the reinforcement learning policy to balance motion stability and fault adaptability in complex damage scenarios through weight allocation;

[0130] (3) The introduction of a heterogeneous actor-critic network and an independent identification network to estimate the robot body structure parameters in real time and encode the damage state as input features enables the control strategy to be dynamically adjusted according to the current working conditions; by online identifying the damage parameters and jointly optimizing the strategy, the generalization ability for different damage modes is significantly improved, and seamless migration from simulation to real scenarios is supported. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0132] Figure 1 It is an overall schematic diagram of the present invention;

[0133] Figure 2 Schematic diagram of the data-driven model. DETAILED DESCRIPTION

[0134] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0135] Example

[0136] The present invention provides a quadruped robot fault-tolerant control method based on residual learning, comprising the following steps:

[0137] Step 1:

[0138] Based on Isaac Gym, a dynamic simulation environment for a quadruped robot under robot limb injury conditions is constructed. The URDF model file of the quadruped robot is modified and loaded into Isaac Gym to simulate the robot limb fracture scenario. An interactive interface between the simulation environment and the algorithm is constructed, where reset(·) is used to initialize the environment and return the initial observation state, and is called at the beginning of each training round. step(·) is the core interactive function, which is used to execute robot actions and return environmental feedback. Its input is the robot joint angle obtained by the fault-tolerant control strategy, and its return value includes the state at the next moment, the immediate reward of the current action, a Boolean value indicating whether the current round is terminated, and debugging information. close(·) is used to close the environment and release resources, and seed(·) is used to set the random seed to ensure the repeatability of the experiment.

[0139] The quadruped robot body mechanism model based on phase information is constructed in this step. The model is not exposed to the user in the form of an independent interface, but is embedded in the step(·) function. Specifically, the motion period of the quadruped robot foot end is defined as T stride , which consists of two stages: the swing stage T sw and support phase T st , and T stride =T sw +T st . Swing phase T swThe duration can be manually specified according to specific circumstances. The time of the support phase is where v d represents the predefined speed, and L span is half of the step length. The phase is designed based on the contact information of the reference leg l ref with the ground. When the force of the reference leg in the z direction exceeds the threshold, it is defined as a touchdown event, represented by the boolean value TD ref indicating this event, which is true when l ref contacts the ground. During the swing phase, the phase of each leg can be calculated using the following formula:

[0140]

[0141] where is the touchdown moment of l ref , represents the time elapsed after i t represents the clock of each leg, and ΔS ref，i represents the phase difference of leg i relative to l ref . The present invention adopts a diagonal gait, in which the phases of the left front leg and the right hind leg are the same, and the phases of the right front leg and the left hind leg are the same. Therefore, when choosing the left front leg or the right hind leg as the reference leg, Similarly, when choosing the right front leg or the left hind leg as the reference leg, as shown in the following formula:

[0142]

[0143] Next, the clock signal is normalized by mapping t i to S i (t). When a leg is in the swing phase, 0 ≤ S i (t) ≤ 1, and when in the support phase, 1 ≤ S i (t) ≤ 2, as follows:

[0144]

[0145] After determining the phase of each leg, it is used to design the end-effector trajectory of the robot. During the swing phase, a Bezier curve is used as the reference trajectory, and during the support phase, a sine curve is used as the reference trajectory, as follows:

[0146]

[0147] s.t. 0 ≤ S i (t) ≤ 1

[0148]

[0149] s.t. 1 ≤ S i (t) ≤ 2

[0150] where is the position of the robot's foot tip relative to the abduction joint coordinate system during the swing phase, and represent its coordinates in three directions respectively, is the k-th control point in the xy plane, is the k-th control point in the z direction, represents the Bernstein polynomial of degree n, and ρ represents the angle of the robot's body forward relative to the target direction. represents the position of the foot tip during the support phase, where σ represents the amplitude of the sine curve. By using the inverse kinematics of the robot, the reference positions of the robot's joints can be determined based on these values as follows:

[0151]

[0152] where IK(·) represents the inverse kinematic solution of the quadruped robot.

[0153] In addition, the reward function for fault-tolerant motion control is also defined in this step. Similarly, this function is not exposed to the user in the form of an independent interface, but is implemented inside the step(·) function. Specifically, the reward function consists of 6 terms. Among them, the body linear velocity reward term r1 aims to encourage the robot to track the target velocity, and its calculation formula is as follows:

[0154]

[0155] where represents the desired linear velocity of the robot's body, v base represents the actual linear velocity, represents the desired Euler angles of the body, and Φ represents the current Euler angles. The body attitude reward term r2 imposes a penalty on the rotation of the robot's body in the roll and pitch directions, which helps to maintain the balance of the robot's body, and its calculation formula is as follows:

[0156]

[0157] The foot velocity reward term r3 encourages the robot's foot to move in the desired direction, and its calculation formula is as follows:

[0158]

[0159] where represents the target velocity of the foot, v feet represents the actual velocity of the foot. The energy loss reward term r4 aims to penalize the motor energy consumption, and its calculation formula is as follows:

[0160]

[0161] where τ represents the torque of each joint, represents the speed of each joint, and t corresponds to the time required for each time step of reinforcement learning. The body contact reward term r5 is used to impose a penalty on the contact between other components of the robot except the feet and the ground, and its calculation formula is as follows:

[0162]

[0163] where TD others is a boolean value used to indicate whether other parts of the robot except the feet are in contact with the ground, and N l represents the total number of components of the robot. The foot-end contact reward term r6 is used to encourage the contact between the robot's feet and the ground, and its calculation formula is as follows:

[0164]

[0165] where TD feet is used to indicate whether the feet are in contact with the ground, w1, w2, and w3 are predefined constants, and the reward value r t for each step of reinforcement learning is the weighted sum of the above six reward values, as follows:

[0166] r t = 1×r1 + 0.5×r2 + 0.2×r3 - 0.1×r4 - 0.1×r5 - 0.1×r6

[0167] Step 2:

[0168] A data-driven model based on a heterogeneous actor-critic architecture is constructed in this step. An actor network, a critic network, and an identification network are respectively constructed based on the PyTorch toolbox. Specifically, the actor network adopts a multi-layer perceptron architecture, and the number of neurons in each layer of the network is set to [N Actor , 512, 256, 128, 12], where N Actor is the dimension of the input information. The inputs of this network include terrain information, proprioceptive information, body structure parameters estimated by the identification network, and body linear velocity information, and the output is the correction amount of the body mechanism model. Among them, the proprioceptive information is directly collected by the sensors on the robot, covering the body angular velocity and gravity projection collected by the IMU; the joint angles and joint angular velocities collected by the joint encoders; and the robot motion instruction information sent by the user. The terrain information is obtained by the lidar installed on the robot. To improve the training efficiency of the model, a heterogeneous critic network is designed, and its network architecture is [N Critic, 512, 256, 128, 1]. Both the actor network and the critic network use the exponential linear unit (ELU) as the activation function between hidden layers. Compared with the actor network, the critic network has richer input information, including privileged information, terrain information, and proprioceptive information. Among them, the privileged information not only includes the real robot body structure parameters and the real body linear velocity, but also includes the foot-end touchdown state of the robot, the external disturbance force, and the gain parameters of the proportional-derivative controller. The actor network and the critic network are optimized by the proximal policy optimization algorithm, and their loss functions are respectively:

[0169]

[0170] where θ is the parameter of the actor network, and φ is the parameter of the critic network. represents the ratio of the new and old policies, and A t is the advantage function, which is calculated by the generalized advantage estimation algorithm. ∈ is the clipping parameter, which is used to limit the change range of r t . When A t > 0, it is used to limit the policy improvement amplitude. When A t < 0, it is used to limit the policy degradation amplitude. V target is the target value function, which is usually calculated by the temporal difference algorithm.

[0171] Considering that the body structures of the robot under different damage conditions are different, an identification network is designed to estimate the body structure parameters of the robot in the current damage state in real time and use them as the input of the actor network, so that it can generate more suitable control strategies according to different structure information. The identification network is also constructed by a multi-layer perceptron, and its network architecture is [N identify , 256, 128, 11]. This network uses the rectified linear unit (ReLU) as the activation function between hidden layers. The input of this network is the historical proprioceptive information, and the output contains two parts: one part is the estimated body structure parameters, and the other part is the estimated body linear velocity. The loss function of this network consists of the regression loss and the loss of the actor network in reinforcement learning in two parts, and the specific form is as follows:

[0172]

[0173] where α is a hyperparameter used to balance different components of the loss function, v t and respectively represent the body linear velocity estimated by the identification network and the actual body linear velocity. and respectively represent the estimated and actual body structure parameters. The body structure parameters are represented by an 8-dimensional vector, and each value corresponds to the structural state of the left front thigh, left front calf, right front thigh, right front calf, left rear thigh, left rear calf, right rear thigh, and right rear calf respectively. A value of 0 indicates that this part has been completely broken, a value of 1 indicates that this part is intact, and a value between 0 and 1 indicates the proportion of the undamaged part in the whole.

[0174] Based on the idea of residual learning, the joint position residual output by the data-driven model is coupled and corrected with the reference joint position generated by the body mechanism model to construct the target joint position command. To suppress high-frequency interference and improve control continuity, a low-pass Butterworth filter is used to smooth the target joint position, and then the smooth joint position is mapped to the target joint torque through a proportional-derivative (PD) tracking controller, and finally the actuator drive signal of the quadruped robot is generated.

[0175] Among them, the dynamic equation of the PD controller can be expressed as:

[0176]

[0177] Among them and represent the desired joint angle and angular velocity, is set to 0. q j and represent the current joint angle and angular velocity respectively, and the gain parameters k p and k d are specified manually.

[0178] Step three:

[0179] Based on the simulation environment constructed in Step 1 and the data-driven algorithm designed in Step 2, large-scale parallel training tasks are carried out on a high-performance server equipped with an Intel(R) Xeon(R) Gold 6226R central processing unit (base frequency 2.90 GHz), 32 GB of memory, and an NVIDIA RTX 3090 graphics card (24 GB of video memory). By synchronously training 4096 quadruped robot agents with different damage configurations, a total of 6000 training iterations are completed. The termination condition for the training round is defined as: the robot's torso or hip joint collides with the ground, or the continuous movement displacement is lower than the threshold for more than 5 seconds. To improve the generalization ability of the policy in complex terrains, a progressive curriculum learning framework is adopted. The training environment is divided into 5 levels according to difficulty, and multi-modal terrains such as smooth ground, rough textures, discretized grids, stepped terrains, and combinations of random obstacles are introduced. At the same time, to alleviate the Sim-to-Real difference, key dynamic parameters are randomly perturbed during the training phase, including motor friction coefficients, proportional-derivative controller gains, motor torque coefficients, and time delays, to enhance the robustness of the policy through domain randomization.

[0180] Step 4:

[0181] After training is completed, the actor network and the identification network are converted from the.pth format to a standard ONNX computation graph through the torch.onnx.export() interface of the PyTorch framework, and the model inference consistency is verified through ONNX Runtime. Further, the NvOnnxParser toolchain of TensorRT is used to parse the ONNX model, construct an inference engine with FP16 precision, and write low-latency inference code based on CUDA 11.6 and the C++17 standard. To achieve simulation-reality control consistency, the ontology mechanism model and the data-driven model are integrated and deployed to the NVIDIA Jetson AGX Xavier embedded platform. Through the real-time API interface provided by the quadruped robot manufacturer, a multi-modal input pipeline is constructed, and the fused control instructions (mechanism model output + data-driven model residual correction) are sent to the underlying motor controller at a frequency of 1 kHz. To eliminate the trajectory jitter caused by discrete control, cubic spline interpolation is used to upsample the 50 Hz policy output to ensure consistency with the simulator step size. Finally, the stability of the fault-tolerant policy is verified in diverse test scenarios, and the algorithm is iteratively optimized according to the test results.

[0182] As Figure 1, which is the overall schematic diagram of the present invention. The reference foot-end trajectory output by the reference trajectory generator is obtained by inverse kinematics calculation to obtain the reference joint positions of the robot. Subsequently, it is superimposed and calculated with the reference joint position residuals generated by the data-driven model to form the target joint position command. This command is smoothed by a low-pass filter and finally generates the target joint torque output through the joint PD controller;

[0183] As Figure 2 , which is the schematic diagram of the data-driven model. The input of the critic network is privileged information, terrain information, and proprioceptive information, and the output is the state value function; the input of the identification network is historical proprioceptive information, and the output is the estimated structural information and the estimated body velocity information; the input of the policy network is terrain information, proprioceptive information, and the estimated structural information and the estimated body velocity information, and the output is the reference joint position residual.

[0184] Those skilled in the art know that in addition to implementing the systems, devices, and their respective modules provided by the present invention in the form of pure computer-readable program codes, the method steps can be logically programmed to enable the systems, devices, and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same program. Therefore, the systems, devices, and their respective modules provided by the present invention can be regarded as a kind of hardware component, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the method or the structures within the hardware component.

[0185] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific implementation manners. Those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined arbitrarily with each other.

Claims

1. A fault-tolerant control method for quadruped robots based on residual learning, characterized in that, Including: Step 1: Construct a mechanism model of the quadruped robot body based on phase information; Step 2: Construct a reward function for fault-tolerant motion control; Step 3: Construct a data-driven model based on a heterogeneous actor-critic architecture; Step 4: Based on the residual learning framework, train and optimize the data-driven model through the reward function, use the obtained residual information as the residual correction amount of the quadruped robot body mechanism model, and finally generate the motion control command of the quadruped robot by coupling the output of the body mechanism model and the residual correction amount.

2. The fault-tolerant control method for a quadruped robot based on residual learning according to claim 1, characterized in that, The said Step 1 includes: The motion cycle of the foot tip of a quadruped robot is defined as T stride , including two phases: the swing phase T sw and the support phase T st , and T stride = T sw + T st ; the duration of the swing phase T sw is manually specified according to specific circumstances, and the time of the support phase is where v d represents the predefined speed, and L span is half of the stride length; The core of the body mechanism model lies in determining the phase of each leg, and the phase is designed based on the contact information of the reference leg l ref with the ground. When the force of the reference leg in the z direction exceeds the threshold, it is defined as a touchdown event, represented by the boolean value TD ref to indicate this event, which is true when l ref contacts the ground; during the swing phase, the phase of each leg is calculated using the following formula: Among them, is the touchdown moment of l ref , and represents the time elapsed after that; t represents the current moment; Adopt a diagonal gait, where the phases of the left front leg and the right hind leg are the same, and the phases of the right front leg and the left hind leg are the same. When choosing the left front leg or the right hind leg as the reference leg, When choosing the right front leg or the left hind leg as the reference leg, As shown in the following formula: where, ΔS ref,i represents the phase difference of leg i relative to l ref ; respectively represent the phase differences of the left front leg FL, the right hind leg HR, the right front leg FR, and the left hind leg HL relative to the reference leg in the diagonal trot gait; ΔS ref，FL , ΔS ref，FR , ΔS ref,HL , ΔS ref,HR respectively represent the phase differences of the left front leg FL, the right front leg FR, the left hind leg HL, and the right hind leg HR relative to the reference leg; Next, normalize the clock signal by mapping the clock \(t\) of each leg \(i\) i to \(S\) i (\(t\)), where \(0\leq S\) i (\(t\))\(\leq1\) when a leg is in the swing phase and \(1\leq S\) i (\(t\))\(\leq2\) when in the stance phase. The normalized clock signal function \(S\) i (\(t\)) is as follows: Among them, T st represents the time period when the foot end of the quadruped robot is in the support phase; T sw represents the time period when the foot end of the quadruped robot is in the swing phase; After determining the phase of each leg, use it to design the end-effector trajectory of the robot. In the swing phase, use the Bezier curve as the reference trajectory, and in the support phase, use the sine curve as the reference trajectory, as follows: s.t. 1 ≤ S i (t) ≤ 2 Among them, is the position of the robot's foot tip relative to the abduction joint coordinate system during the swing phase, and respectively represent its coordinates in three directions, is the k-th control point in the xy plane, is the k-th control point in the z direction, represents the Bernstein polynomial of degree n, where n is the number of control points, and ρ represents the angle of the positive direction of the robot's body relative to the target direction, represents the position of the foot tip during the support phase, and σ represents the amplitude of the sine curve; represents the coordinates of the foot tip in three directions during the support phase; Using robotic inverse kinematics, the reference positions q of the robot joints are determined based on these values ref , as follows: Where, IK(·) represents the inverse kinematics of the quadruped robot.

3. The fault-tolerant control method for a quadruped robot based on residual learning according to claim 2, characterized in that, The said Step 2 includes: The reward function contains 6 terms. Among them, the body linear velocity reward term r1 aims to encourage the robot to track the target velocity, and its calculation formula is as follows: Among them, represents the desired linear velocity of the robot body, represents the actual linear velocity, represents the desired Euler angles of the body, Φ z represents the current Euler angles; tanh is the hyperbolic tangent function; k1 is the dynamic weight coefficient based on the attitude error, used to adjust the value of the reward function r1; The body attitude reward term r2 imposes a penalty on the rotation of the robot body in the roll and pitch directions to maintain the balance of the robot body, and its calculation formula is as follows: Among them, represents the square of the roll angle Φ of the robot x and the pitch angle Φ y squared; The foot velocity reward term r3 encourages the robot's feet to move in the desired direction, and its calculation formula is as follows: Among them, represents the target speed of the foot, represents the actual speed of the foot, where w1, w2, and w3 are predefined constants; k3 is a dynamic weight coefficient based on the attitude error, used to adjust the value of the reward function r3; The energy loss reward term r4 aims to penalize the motor energy consumption, and its calculation formula is as follows: Among them, τ i represents the torque of each joint, represents the speed of each joint, and t corresponds to the time required for each time step of reinforcement learning; The body contact reward term r5 is used to impose a penalty on the contact between other components of the robot except the feet and the ground, and its calculation formula is as follows: Among them, is a boolean value used to indicate whether parts of the robot other than the feet are in contact with the ground; N l represents the total number of components of the robot; The end-effector contact reward term r6 is used to encourage the contact between the robot's feet and the ground, and its calculation formula is as follows: Among them, used to indicate whether the foot contacts the ground; The reward value r for each step of reinforcement learning t is the weighted sum of the above six reward values, as follows: r t ＝1×r1+0.5×r2+0.2×r3-0.1×r4-0.1×r5-0.1×r6。 4. The fault-tolerant control method for a quadruped robot based on residual learning according to claim 1, characterized in that The said Step 3 includes: The data-driven model consists of three parts: an actor network, a critic network, and an identification network; among them, the actor network adopts a multi-layer perceptron architecture, and the number of neurons in each layer of the network is set to [N Actor , 512, 256, 128, 12], where N Actor is the dimension of the input information. The input of this network includes terrain information, body perception information, the body structure parameters estimated by the identification network, and the body linear velocity information, and the output is the correction amount of the body mechanism model; among them, the body perception information is directly collected by the sensors on the robot, covering the body angular velocity and gravity projection collected by the IMU, the joint angles and joint angular velocities collected by the joint encoders, and the robot motion instruction information sent by the user; the terrain information is obtained by the lidar installed on the robot; design a heterogeneous critic network to improve the training efficiency of the model, and its network architecture is [N Critic , 512, 256, 128, 1]; N Critic represents the dimension of the input information of the Critic network; Both the actor network and the critic network use the exponential linear unit as the activation function between hidden layers. The input of the critic network includes privileged information, terrain information, and body perception information. Among them, the privileged information not only includes the real robot body structure parameters and the real body linear velocity, but also includes the foot end touchdown state of the robot, the external disturbance force, and the gain parameters of the proportional-derivative controller; the actor network and the critic network are optimized through the proximal policy optimization algorithm, and their loss functions are respectively: where θ are the parameters of the actor network and φ are the parameters of the critic network; represents the ratio of the new and old policies; π θ denotes the policy currently being optimized, denotes the old policy previously used to collect experience data; a t denotes the action selected by the agent at time step t; s t denotes the environmental state at time step t; denotes the expectation for the current batch of samples; V φ (s t ) denotes the value estimate of state s t by the Critic network; A t is the advantage function, calculated by the Generalized Advantage Estimation algorithm; ∈ is the clipping parameter, used to limit the t range of change of r; when A t > 0, it is used to limit the policy improvement amplitude, and when A t < 0, it is used to limit the policy degradation amplitude; A target is the target value function, calculated using the Temporal Difference algorithm; Considering that the body structure of the robot varies under different damage conditions, a recognition network is designed to estimate the body structure parameters of the robot in the current damage state in real time and use them as the input of the actor network; the recognition network is constructed by a multi-layer perceptron, and its network architecture is [N identify , 256, 128, 11]; N identify represents the dimension of the input information of the recognition network; The network uses the rectified linear unit as the activation function between hidden layers. The input of the network is historical body perception information, and the output consists of two parts: one part is the estimated body structure parameters, and the other part is the estimated body linear velocity; the loss function of the network consists of a regression loss and the loss of the actor network in reinforcement learning and is composed of two parts. The specific form is as follows: where α is a hyperparameter used to balance different components of the loss function; v t and respectively represent the body linear velocity estimated by the identification network and the actual body linear velocity; and respectively represent the estimated and actual body structure parameters; MSE() represents the mean squared error loss function; The body structure parameters are represented by an 8-dimensional vector, and each value corresponds to the structural state of the left front thigh, left front calf, right front thigh, right front calf, left rear thigh, left rear calf, right rear thigh, and right rear calf respectively. A value of 0 means that this part has been completely broken, a value of 1 means that this part is intact, and a value between 0 and 1 means the proportion of the undamaged part in the whole; Based on the residual learning idea, couple and correct the joint position residual output by the data-driven model and the reference joint position generated by the body mechanism model to construct the target joint position command; to suppress high-frequency interference and improve control continuity, use a low-pass Butterworth filter to smooth the target joint position, and then map the smoothed joint position to the target joint torque through a proportional-derivative (PD) tracking controller, and finally generate the actuator drive signal of the quadruped robot. Among them, the dynamic equation of the PD controller is expressed as: wherein, and represent the desired joint angle and angular velocity, are set to 0; q j and represent the current joint angle and angular velocity respectively; the gain parameters k p and k d are specified manually.

5. The fault-tolerant control method for a quadruped robot based on residual learning according to claim 1, wherein Step 4 includes: After training is completed, use the torch.onnx.export() interface of the PyTorch framework to convert the actor network and the recognition network from the.pth format to a standard ONNX computation graph, and verify the model inference consistency through ONNX Runtime; use the NvOnnxParser toolchain of TensorRT to parse the ONNX model, build an inference engine with FP16 precision, and write low-latency inference code based on CUDA 11.6 and the C++17 standard; to achieve simulation-reality control consistency, integrate the body mechanism model and the data-driven model, deploy it to the NVIDIA Jetson AGX Xavier embedded platform, build a multi-modal input pipeline through the real-time API interface provided by the quadruped robot manufacturer, and send the fused control instructions to the underlying motor controller at a frequency of 1 kHz; to eliminate the trajectory jitter caused by discrete control, use cubic spline interpolation to upsample the 50 Hz policy output to ensure consistency with the simulator step size; finally, verify the stability of the fault-tolerant strategy in diverse test scenarios and iteratively optimize the algorithm according to the test results.

6. A fault-tolerant control system for a quadruped robot based on residual learning, characterized in that, including: Module M1: Build a body mechanism model of a quadruped robot based on phase information; Module M2: Build a reward function for fault-tolerant motion control; Module M3: Build a data-driven model based on a heterogeneous actor-critic architecture; Module M4: Based on the residual learning framework, train and optimize the data-driven model through the reward function, use the obtained residual information as the residual correction amount of the body mechanism model of the quadruped robot, and finally generate the motion control instructions of the quadruped robot by coupling the output of the body mechanism model and the residual correction amount.

7. The fault-tolerant control system of the quadruped robot based on residual learning according to claim 6, characterized in that, The said Module M1 includes: The motion cycle of the foot tip of a quadruped robot is defined as T stride , including two phases: the swing phase T sw and the support phase T st , and T stride = T sw + T st ; the duration of the swing phase T sw is manually specified according to the specific situation, and the time of the support phase is where v d represents the predefined speed, and L span is half of the stride length; The core of the body mechanism model lies in determining the phase of each leg, and the phase is designed based on the contact information of the reference leg l ref with the ground. When the force of the reference leg in the z-direction exceeds the threshold, it is defined as a touchdown event, represented by the boolean value TD ref to indicate this event, which is true when l ref contacts the ground; during the swing phase, the phase of each leg is calculated using the following formula: Among them, is the touchdown moment of l ref , and represents the time elapsed after that; t represents the current moment; Adopt a diagonal gait, where the phases of the left front leg and the right hind leg are the same, and the phases of the right front leg and the left hind leg are the same. When choosing the left front leg or the right hind leg as the reference leg, When choosing the right front leg or the left hind leg as the reference leg, As shown in the following formula: Among them, ΔS ref，i represents the phase difference of leg i relative to l ref ; respectively represent the phase differences of the left front leg FL, right hind leg HR, right front leg FR, and left hind leg HL relative to the reference leg in the diagonal trot gait; ΔS ref，FL , ΔS ref，FR , ΔS ref，HL , ΔS ref，HR respectively represent the phase differences of the left front leg FL, right front leg FR, left hind leg HL, and right hind leg HR relative to the reference leg; Next, normalize the clock signal by mapping the clock t of each leg i i to S i (t). When a leg is in the swing phase, 0 ≤ S i (t) ≤ 1, and when in the stance phase, 1 ≤ S i (t) ≤ 2. The normalized clock signal function S i (t) is as follows: Among them, T st represents the time period when the foot tip of the quadruped robot is in the support phase; T sw represents the time period when the foot tip of the quadruped robot is in the swing phase; After determining the phase of each leg, use it to design the foot trajectory of the robot. In the swing phase, use a Bezier curve as the reference trajectory, and in the support phase, use a sine curve as the reference trajectory, as follows: Among them, is the position of the robot's foot tip relative to the abduction joint coordinate system during the swing phase, and respectively represent its coordinates in three directions, is the k-th control point in the xy plane, is the k-th control point in the z direction, represents the Bernstein polynomial of degree n, where n is the number of control points, and ρ represents the angle of the robot's body positive direction relative to the target direction, represents the position of the foot tip during the support phase, and σ represents the amplitude of the sine curve; represents the coordinates of the foot tip in three directions during the support phase; By using robotic inverse kinematics, the reference positions q of the robotic joints are determined based on these values ref , as follows: where IK(·) represents the inverse kinematics of the quadruped robot.

8. The fault-tolerant control system for a quadruped robot based on residual learning according to claim 7, characterized in that The said Module M2 includes: The reward function contains 6 terms. Among them, the body linear velocity reward term r1 aims to encourage the robot to track the target velocity, and its calculation formula is as follows: Among them, represents the desired linear velocity of the robot body, represents the actual linear velocity, represents the desired Euler angles of the body, Φ z represents the current Euler angles; tanh is the hyperbolic tangent function; k1 is the dynamic weight coefficient based on the attitude error, which is used to adjust the value of the reward function r1; The body attitude reward term r2 imposes a penalty on the rotation of the robot's body in the roll and pitch directions to maintain the balance of the robot's body, and its calculation formula is as follows: Among them, represents the square of the roll angle Φ of the robot x and the pitch angle Φ y squared; The foot velocity reward term r3 encourages the robot's feet to move in the desired direction, and its calculation formula is as follows: Among them, represents the target speed of the foot, represents the actual speed of the foot, w1, w2, and w3 are predefined constants; k3 is a dynamic weight coefficient based on the attitude error, used to adjust the value of the reward function r3; The energy loss reward term r4 aims to penalize the motor energy consumption, and its calculation formula is as follows: where τ i represents the torque of each joint, represents the speed of each joint, and t corresponds to the time required for each time step of the reinforcement learning; The body contact reward term r5 is used to impose a penalty on the contact between other components of the robot except the feet and the ground, and its calculation formula is as follows: Among them, is a boolean value used to indicate whether parts of the robot other than the feet are in contact with the ground; N l represents the total number of components of the robot; The foot end contact reward term r6 is used to encourage the contact between the robot's feet and the ground, and its calculation formula is as follows: Among them, used to indicate whether the foot touches the ground; The reward value r for each step of reinforcement learning t is the weighted sum of the above six reward values, as follows: r t ＝1×r1+0.5×r2+0.2×r3-0.1×r4-0.1×r5-0.1×r6。 9. The fault-tolerant control system of a quadruped robot based on residual learning according to claim 6, wherein The said Module M3 includes: The data-driven model consists of three parts: an actor network, a critic network, and an identification network. The actor network adopts a multi-layer perceptron architecture, and the number of neurons in each layer of the network is set to [N Actor , 512, 256, 128, 12], where N Actor is the dimension of the input information. The input of this network includes terrain information, body perception information, the body structure parameters estimated by the identification network, and the body linear velocity information. The output is the correction amount of the body mechanism model. The body perception information is directly collected by the sensors on the robot, covering the body angular velocity and gravity projection collected by the IMU, the joint angles and joint angular velocities collected by the joint encoders, and the robot motion instruction information sent by the user. The terrain information is obtained by the lidar installed on the robot. A heterogeneous critic network is designed to improve the training efficiency of the model, and its network architecture is [N Critic , 512, 256, 128, 1]; N Critic represents the dimension of the input information of the Critic network. Both the actor network and the critic network use the exponential linear unit as the activation function between hidden layers. The inputs of the critic network include privileged information, terrain information, and proprioceptive information. The privileged information not only includes the real robot body structure parameters and the real body linear velocity, but also the foot-end touchdown state of the robot, the external disturbance force, and the gain parameters of the proportional-derivative controller. The actor network and the critic network are optimized by the proximal policy optimization algorithm, and their loss functions are respectively: Among them, θ is the parameter of the actor network, and φ is the parameter of the critic network; represents the ratio of the new and old policies; π θ represents the policy being optimized currently, represents the old policy used to collect experience data before; a t represents the action selected by the agent at time step t; s t represents the environmental state at time step t; represents the expectation for the current batch of samples; V φ (s t ) represents the value estimation of state s t by the Critic network; A t is the advantage function, calculated by the generalized advantage estimation algorithm; ∈ is the clipping parameter, used to limit the t change range of r; when A t > 0, it is used to limit the policy improvement amplitude, and when A t < 0, it is used to limit the policy degradation amplitude; V target is the target value function, calculated by the temporal difference algorithm; Considering that the body structure of the robot varies under different damage conditions, an identification network is designed to estimate the body structure parameters of the robot in the current damage state in real time and use them as the input of the actor network. The identification network is constructed using a multi-layer perceptron, and its network architecture is [N indentify , 256, 128, 11]; N identify represents the dimension of the input information of the identification network; The network uses the rectified linear unit as the activation function between hidden layers. The input of the network is historical body perception information, and the output consists of two parts: one part is the estimated body structure parameters, and the other part is the estimated body linear velocity; the loss function of the network consists of the regression loss and the actor network loss in reinforcement learning and is composed of two parts, and the specific form is as follows: where α is a hyperparameter used to balance different components of the loss function; v t and represent the body linear velocity estimated by the identification network and the actual body linear velocity, respectively; and represent the estimated and actual body structure parameters, respectively; MSE() represents the mean square error loss function; The body structure parameters are represented by an 8-dimensional vector. Each value corresponds to the structural state of the left front thigh, left front calf, right front thigh, right front calf, left rear thigh, left rear calf, right rear thigh, and right rear calf respectively. A value of 0 indicates that this part is completely broken, a value of 1 indicates that this part is intact, and a value between 0 and 1 indicates the proportion of the undamaged part to the whole; Based on the residual learning idea, the joint position residuals output by the data-driven model are coupled and corrected with the reference joint positions generated by the ontological mechanism model to construct the target joint position command. To suppress high-frequency interference and improve control continuity, a low-pass Butterworth filter is used to smooth the target joint position, and then the smooth joint position is mapped to the target joint torque through a proportional-derivative (PD) tracking controller, and finally the actuator drive signal of the quadruped robot is generated. Among them, the dynamic equation of the PD controller is expressed as: wherein, and represent the desired joint angle and angular velocity, is set to 0; q j and represent the current joint angle and angular velocity respectively; the gain parameters k p and k d are specified manually.

10. The fault-tolerant control system of a quadruped robot based on residual learning according to claim 6, characterized in that, The module M4 includes: After training, the actor network and the identification network are converted from the.pth format to a standard ONNX computation graph through the torch.onnx.export() interface based on the PyTorch framework, and the model inference consistency is verified through ONNX Runtime; The ONNX model is parsed using the NvOnnxParser toolchain of TensorRT to construct an inference engine with FP16 precision, and low-latency inference code is written based on CUDA 11.6 and the C++17 standard; To achieve simulation-reality control consistency, the ontological mechanism model and the data-driven model are integrated and deployed to the NVIDIA Jetson AGX Xavier embedded platform. Through the real-time API interface provided by the quadruped robot manufacturer, a multi-modal input pipeline is constructed, and the fused control instructions are sent to the underlying motor controller at a frequency of 1 kHz; To eliminate the trajectory jitter caused by discrete control, cubic spline interpolation is used to upsample the 50 Hz policy output to ensure consistency with the simulator step size; Finally, the stability of the fault-tolerant strategy is verified in diverse test scenarios, and the algorithm is iteratively optimized according to the test results.

Images