A fault-tolerant gait planning method for hexapod robots based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's six-foot robot fault-tolerant gait planning method, comprising the following steps: build the simulation model of six-foot robot;Establish the CPG gait generator based on improved Hopf oscillator;Fusion simulation model, strategy network and the CPG gait generator based on improved Hopf oscillator, construct reinforcement learning motion control framework, for the established six-foot robot simulation model, fusion reinforcement learning motion control framework;Simulate six-foot robot partial leg failure, freeze six-foot robot leg failure in simulation environment, train reinforcement learning motion control framework;The control network after training is integrated in the gait control framework of six-foot robot, for generating the fault-tolerant gait of six-foot robot and verifying, if it can be completed autonomous motion under the condition of freezing failure leg then it indicates that strategy network is effective, to extract strategy network for controlling the motion of real six-foot robot.

Description

Technical Field

[0001] This invention relates to the field of robot control, and more particularly to a fault-tolerant gait planning method for a hexapod robot based on reinforcement learning. Background Technology

[0002] With the development of science and technology, bionic robot technology has rapidly advanced and played a significant role in fields such as medicine, industry, military, and aerospace. Among the many types of robots, hexapods have more advantages in unstructured terrain, including adaptability and flexibility to irregular terrain. Therefore, hexapods have broader application prospects in some special environments. However, when robots move in dangerous or disaster-prone environments, leg failures are prone to occur and cannot be repaired manually in a timely manner. If a gait that allows the robot to continue moving despite leg failures can be found based on the current situation, it is considered fault-tolerant to the given failure. This will improve the applicability of hexapods in unknown environments. Therefore, fault-tolerant control for hexapods is particularly important, ensuring that the robot continues to operate rather than the mission fails completely.

[0003] To address the problem of fault-tolerant gait planning for hexapods, current research mainly explores switching to a fixed fault-tolerant gait to adapt to leg failures, such as a fault-tolerant gait control method for a legless hexapod with missing legs (CN109696824B). Alternatively, it studies the design of an adaptive fault-tolerant gait generator based on hierarchical modeling of the CPG controller, which extends or shortens the support phase according to changes in leg load to generate various gaits, such as (You Bo, Li Kunpeng, Li Jiayu, Liu Daquan. Instability adjustment and fault-tolerant gait design of a hexapod with single-leg failure [J]. Journal of Mechanical Engineering, 2021, 57(01): 100-109.). These methods all require significant human intervention to adjust parameters to achieve the optimal gait of the hexapod, which is time-consuming, laborious, and may not necessarily achieve the best results. Reinforcement learning, as an emerging algorithm, uses a reward function mechanism to find a parameter update strategy with high reward through continuous trial and error and iteration. It can be used to solve the problem of generating fault-tolerant gait after a leg failure in a hexapod robot. By designing an appropriate reward function, a suitable gait output can be found. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a fault-tolerant gait planning method for hexapod robots based on reinforcement learning. In the event of leg failure in a hexapod robot, reinforcement learning is used to find a suitable fault-tolerant gait output, reducing manual parameter tuning intervention and solving the fault-tolerant gait planning problem in the case of leg failure in hexapod robots.

[0005] The present invention is achieved by at least one of the following technical solutions.

[0006] A reinforcement learning-based fault-tolerant gait planning method for a hexapod robot includes the following steps:

[0007] S1. Build a simulation model of the hexapod robot;

[0008] S2. Based on the motion characteristics of the hexapod robot, a CPG gait generator based on an improved Hopf oscillator is established. The input of the gait generator is the gait parameters, and the output is the joint position control command of the hexapod robot, so as to control the robot to move according to the gait generated by the gait generator.

[0009] S3. Integrate the simulation model with the CPG gait generator based on the improved Hopf oscillator in step S2 to construct a reinforcement learning motion control framework;

[0010] S4. Simulate random failures of some legs of a hexapod robot. In the simulation environment, the failed leg of the hexapod robot is set to be unable to move and without support. Train the reinforcement learning motion control framework to obtain the parameters of the control network, so that the framework can control the simulation model of the hexapod robot to move in the simulation scene after some legs fail.

[0011] S5. Integrate the trained control network into the gait control framework of the hexapod robot to generate the fault-tolerant gait of the hexapod robot, and verify it in a simulation environment. If the robot can complete the movement in the event of leg failure, it means that the policy network is effective. Thus, the policy network can be extracted to control the movement of the real hexapod robot.

[0012] Furthermore, in step S2, the mathematical model of the improved Hopf oscillator is as follows:

[0013]

[0014] In the formula, ω is the frequency of the oscillator; ω stance It is the supporting phase frequency; ω swing y is the oscillation phase frequency; b is a constant; β is the occupancy factor; y is the state variable of the oscillator.

[0015] Furthermore, in step S2, the mathematical model of the oscillators of the six legs of the hexapod robot, which are coupled together to form a ring-coupled network CPG gait generator, is as follows:

[0016]

[0017] In the formula: λ is the coupling strength parameter between the two oscillators; x i and y i x is the state variable of oscillator i; j and y j It is the state variable of oscillator j; and It is the first derivative; α is the convergence rate coefficient; μ is the square of the oscillator amplitude; ω i θ is the frequency of a single oscillator. ji It is the phase difference between oscillators i and j; ω stance It is the supporting phase frequency; ω swing is the oscillation phase frequency; b is a constant.

[0018] Furthermore, in step S2, the mapping function between the hip joint, knee joint, and ankle joint and the output curve of the oscillator is:

[0019]

[0020] In the formula: θ1, θ2, and θ3 are the rotation angles of the hip, knee, and ankle joints, respectively; k0 is the mapping coefficient of the hip joint; k1 and k2 are the mapping coefficients of the knee joint; k3 is the mapping coefficient of the ankle joint, which is used to adjust the amplitude of the joint control signal; and x and y are the state variables of the oscillator.

[0021] Furthermore, the reinforcement learning motion control framework includes:

[0022] Define the state variable S of the hexapod robot in the simulation environment, wherein the state variable S includes the pitch angle θ of the robot platform. pitch Roll angle θ roll The linear velocity v and angular velocity ω of the body platform, and the rotation angle θ of each joint. i ;

[0023] Define the action variable A for a CPG gait generator based on an improved Hopf oscillator, the action variable including the footprint coefficient β and the phase difference θ between the individual oscillators i and j. ji ;

[0024] A control network for a hexapod robot is constructed, comprising a policy network, a total state value function network, and two action state value function networks; each network structure is a neural network structure, including an input layer, a hidden layer, and an output layer.

[0025] Furthermore, the reward function of the policy network includes forward distance, body roll degree, energy efficiency, and joint angle mutation, expressed as:

[0026]

[0027] In the formula, r is the reward function, and θ pitch Let θ be the pitch angle. roll Let Δt be the roll angle, Δt be the time difference, d be the robot's forward direction, and x be the roll angle. t -x t-1Let τ be the distance traveled from time t-1 to time t. n For joint torque, Let θ be the joint velocity. it -θ i(t-1) λ1, λ2, λ3, and λ4 represent the abrupt change from joint t-1 to time t, where λ1, λ2, λ3, and λ4 are user-defined coefficients.

[0028] Furthermore, the policy network comprises a four-layer neural network structure: an input layer, two hidden layers, and an output layer, which maps the state space to the action space.

[0029] Furthermore, the total state value function network includes an online state value function network and a target state value function network, which have the same structure.

[0030] Furthermore, the action state value function network is a network that maps the state space to the action state value space, and the two action state value function networks have the same structure.

[0031] Furthermore, the parameter update of the action state value function network uses MSELoss as the loss function.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] This invention employs a model-free reinforcement learning algorithm to plan the fault-tolerant gait of a hexapod robot after partial leg failure. It automatically generates the input parameters of the CPG gait generator to generate the fault-tolerant gait, replacing the traditional method of manually designing the controller for fault-tolerant gait planning and reducing the workload of manually adjusting parameters. Attached Figure Description

[0034] Figure 1 This is a model diagram of the hexapod robot in an embodiment of the present invention;

[0035] Figure 2 This is a reinforcement learning motion control framework according to an embodiment of the present invention;

[0036] Figure 3 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0037] This section will describe specific embodiments of the present invention in detail. Preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, enabling a person to intuitively and vividly understand the overall technical solution of the present invention, but they should not be construed as limiting the scope of protection of the present invention. The step numbers in the following embodiments are only set for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0038] like Figure 3 As shown, this embodiment provides a fault-tolerant gait planning method for a hexapod robot based on reinforcement learning, including the following steps:

[0039] S1. Build a simulation model of the six-legged robot.

[0040] Based on the robot's 3D model, a simplified robot simulation model is constructed, enabling the model to perform 3D simulation motion in a simulation environment. This includes the following steps:

[0041] S12, such as Figure 1 As shown, a simplified 3D model of a hexapod robot assembly is constructed based on the existing model. The hexapod robot model is divided into a body platform and six legs, with each leg consisting of three links. Specifically, the body platform of the hexapod robot is rectangular, and the connection points of each leg are at the four vertices of the rectangle and the midpoint of its long side. Each leg of the hexapod robot contains three active joints: the hip joint, the knee joint, and the ankle joint. The hip joint can rotate around an axis perpendicular to the plane of the body, while the rotation axes of the other two joints are perpendicular to the rotation axis of the hip joint.

[0042] S13. Convert the 3D model into an STL model and write the URDF model file for the hexapod robot, setting parameters such as joint mass, link mass, and link moment of inertia.

[0043] S14. Import the simulation model into the Pybullet simulation environment, import the ground model, and set the gravity acceleration. Write a program to test that the joints of the hexapod robot simulation model can normally receive position control commands, complete the test of the simulation model, and ensure that motion simulation can be performed normally.

[0044] S2. Based on the motion characteristics of the hexapod robot, a CPG gait generator based on an improved Hopf oscillator is established. The input of the gait generator is the gait parameters, and the output is the joint position control command of the hexapod robot, so as to control the robot to move according to the gait generated by the gait generator.

[0045] The Hopf oscillator, as an oscillation unit in a CPG network, has the following mathematical expression:

[0046]

[0047] In the formula: x and y are the state variables of the oscillator. ω is the first derivative, i.e., the output of the oscillator; α is the convergence rate coefficient; μ is the square of the oscillator amplitude; ω is the frequency of the oscillator.

[0048] In the original Hopf oscillator's output waveform, the rise and fall phases of the signal are of equal duration within a single cycle, making it only applicable to triangular gait. Here, a gait factor β is added to the original oscillator model, and the relationship is:

[0049]

[0050] In the formula, ω is the frequency of the oscillator; ω stance It is the supporting phase frequency; ω swing y is the oscillation phase frequency; b is a constant; β is the footprint coefficient; y is the state variable of the oscillator;

[0051] When β = 0.5, the durations of the oscillating phase and the supporting phase are the same. Changing the value of β can adjust the durations of the oscillating phase and the supporting phase.

[0052] The oscillators of each leg of the hexapod robot are coupled together, continuously outputting joint angle control signals, enabling the hexapod robot to move in various gaits. A ring-coupled network topology is used to describe the phase coupling relationship between the output signals of each oscillator model. The mathematical model of the ring-coupled network CPG gait generator is as follows:

[0053]

[0054] In the formula: λ is the coupling strength parameter between the two oscillators; x i and y i x is the state variable of oscillator i; j and y j It is the state variable of oscillator j; and It is the first derivative; α is the convergence rate coefficient; μ is the square of the oscillator amplitude; ω i θ is the frequency of a single oscillator. ji It is the phase difference between oscillators i and j; ω stance It is the supporting phase frequency; ω swing is the oscillation phase frequency; b is a constant.

[0055] Since the oscillator's output signal is dimensionless and cannot be directly used as a joint control signal, a mapping function method is used to transform the model's output from a dimensionless quantity into a joint angle control quantity. The angles of the hip, knee, and ankle joints are θ1, θ2, and θ3, respectively. The mapping function between them and the oscillator's output curve is as follows:

[0056]

[0057] Where: k0 is the mapping coefficient of the hip joint; k1 and k2 are the mapping coefficients of the knee joint; k3 is the mapping coefficient of the ankle joint, used to adjust the amplitude of the joint control signal.

[0058] As a preferred embodiment, considering the mechanical structure and parameters of the hexapod robot, k0 = 0.5, k1 = 0.33, k2 = 0.15, and k3 = -0.18;

[0059] S3, such as Figure 2 As shown, the simulation model is integrated with the CPG gait generator based on the improved Hopf oscillator in step S2 to construct a reinforcement learning motion control framework and set training rules, specifically including the following steps:

[0060] S31. Define the state variable S of the hexapod robot in the simulation environment. The state variable S includes the pitch angle θ of the robot platform. pitch Roll angle θ roll The linear velocity v and angular velocity ω of the body platform, and the rotation angle θ of each joint. i ;

[0061] S32. Define the action variable A for a CPG gait generator based on an improved Hopf oscillator, wherein the action variable includes the footprint coefficient β and the phase difference θ between each oscillator i and j. ji ;

[0062] S33. Construct the structure of the reinforcement learning control network for a hexapod robot, wherein the control network includes a policy network, an online state value function network, a target state value function network, and two action state value function networks;

[0063] S34. Define the reward function for the fault-tolerant gait strategy, wherein the reward function of the fault-tolerant gait strategy consists of four parts: forward distance, body roll degree, energy efficiency, and joint angle mutation, and the expression is:

[0064]

[0065] In the formula, r is the reward function, and θ pitch Let θ be the pitch angle. roll Let Δt be the roll angle, Δt be the time difference, d be the robot's forward direction, and x be the roll angle. t -x t-1 Let τ be the distance traveled from time t-1 to time t. n For joint torque, Let θ be the joint velocity. it -θ i(t-1) λ1, λ2, λ3, and λ4 represent the abrupt change from joint t-1 to time t, where λ1, λ2, λ3, and λ4 are user-defined coefficients.

[0066] S4. Simulate random failures of some legs of a hexapod robot. In the simulation environment, the failed leg of the hexapod robot is set to be unable to move and without support. Train the reinforcement learning motion control framework to obtain the parameters of the control network, so that the framework can control the simulation model of the hexapod robot to move in the simulation scene after some legs fail.

[0067] The control network structure of the reinforcement learning motion control framework includes a policy network, a total state value function network, and two action state value function networks. The policy network, after training, is integrated into the fault-tolerant gait control framework of the hexapod robot, while the other networks serve as auxiliary training networks. All network structures are neural network structures, containing input layers, hidden layers, and output layers.

[0068] As a preferred embodiment, the specific settings of each network structure are as follows:

[0069] The policy network has a four-layer neural network structure that maps the state space to the action space. The input layer of this network structure has 32 nodes, corresponding to the defined state variables; the hidden layers have 256 and 128 nodes respectively; and the output layer has 5 nodes, corresponding to the defined action variables.

[0070] The total state value function network includes an online state value function network and a target state value function network, both with identical structures. The input layer of this neural network has 37 nodes, containing state and action variables; the hidden layers have 256 and 128 nodes respectively; and the output layer has one node, corresponding to the estimated state value.

[0071] The action-state-value function network maps the state space to the action-state-value space, and the two action-state-value function networks have the same structure. The input layer of this network structure has 32 nodes, representing state variables; the hidden layers have 256 and 128 nodes respectively; and the output layer has 5 nodes, corresponding to the estimated values ​​of the action states.

[0072] The specific training process is as follows:

[0073] S41. Initialize the control network parameters. The initialization parameters for the two state value function networks are the same, and the initialization parameters for the two action state value function networks are also the same. The control network structure uses a four-layer neural network, including an input layer, two hidden layers, and an output layer.

[0074] S42. Randomly initialize the simulation environment, including the robot's posture, link parameters, and terrain environment, and obtain the robot's initial state S. tSpecifically, this embodiment employs a parameter randomization method, including the initial pose, link mass, and moment of inertia of the hexapod robot. The initial parameters are values ​​calculated using 3D modeling software, and a random parameter is sampled from the uniform distribution of these values.

[0075] S43. Initialize the robot's state S t Input the control strategy network to obtain the output action value A. t The motion parameters of the robot are obtained by outputting the motion to the gait generator, and the simulated robot is controlled to complete one cycle of gait movement, thus obtaining another state value S. t+1 And obtain the reward value R for this step according to the reward function. t , will (S t A t R t S t+1 Store it in the experience pool.

[0076] S44. Randomly select n data points from the experience pool as a mini-batch, calculate the gradient of the online state value function network, and use the Adam algorithm to update the parameters.

[0077] S45. Again, randomly select a small batch of data from the experience pool to update the network parameters of the action-state value function. State S t The true value is estimated to be V s Using actual action A t The obtained Q(S) t A t The value is used as the predicted value estimate of the state, and MSELoss is used as the loss function to train the action state value function network.

[0078] S46. Update the policy network parameters based on the reward value.

[0079] S47. Perform soft updates on the network parameters of the target state value function.

[0080] S48. Repeat steps S42 to S47 until the policy network converges.

[0081] S5. Integrate the trained policy network into the gait control framework of the hexapod robot to output gait parameters to control the fault-tolerant gait output by the CPG network, and verify it in a simulation environment. If the movement can be completed when the hexapod robot has leg failure, it means that the policy network is effective. Thus, the policy network can be extracted to control the movement of the real hexapod robot.

[0082] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A fault-tolerant gait planning method for a hexapod robot based on reinforcement learning, characterized in that, Includes the following steps: S1. Build a simulation model of the hexapod robot; S2. Based on the motion characteristics of the hexapod robot, a CPG gait generator based on an improved Hopf oscillator is established. The input of the gait generator is the gait parameters, and the output is the joint position control command of the hexapod robot, so as to control the robot to move according to the gait generated by the gait generator. The mathematical model of the improved Hopf oscillator is as follows: In the formula, The frequency of the oscillator; It supports the phase frequency; It is the oscillation phase frequency; b is a constant; It is the footprint coefficient; y is the state variable of the oscillator; The mathematical model for the CPG gait generator, which consists of interconnected oscillators on the six legs of a hexapod robot, is as follows: In the formula: It is the coupling strength parameter between the two oscillators; and It is an oscillator i State variables; and It is an oscillator j State variables; and It is the first derivative; It is the convergence rate coefficient; The square of the oscillator amplitude; The frequency of a single oscillator; It is an oscillator i and j The phase difference between them; It supports the phase frequency; It is the oscillation phase frequency; b It is a constant; S3. Integrate the simulation model with the CPG gait generator based on the improved Hopf oscillator in step S2 to construct a reinforcement learning motion control framework; S4. Simulate random failures in some legs of a hexapod robot. In the simulation environment, the faulty leg of the hexapod robot is set to be unable to move and without support. The reinforcement learning motion control framework is trained to obtain the parameters of the control network, enabling the framework to control the motion of a simulation model of a hexapod robot in a simulation scenario after some legs fail. S5. Integrate the trained control network into the gait control framework of the hexapod robot to generate the fault-tolerant gait of the hexapod robot, and verify it in a simulation environment. If the robot can complete the movement in the event of leg failure, it means that the policy network is effective. Thus, the policy network can be extracted to control the movement of the real hexapod robot.

2. The reinforcement learning-based fault-tolerant gait planning method for a hexapod robot according to claim 1, characterized in that, In step S2, the mapping function between the hip joint, knee joint, and ankle joint and the output curve of the oscillator is: In the formula: , , These refer to the angles of rotation at the hip, knee, and ankle joints, respectively. It is the mapping coefficient of the hip joint; , It is the mapping coefficient of the knee joint; It is the ankle joint mapping coefficient, used to adjust the amplitude of the joint control signal. x, y For the state variables of the oscillator.

3. The reinforcement learning-based fault-tolerant gait planning method for a hexapod robot according to claim 1, characterized in that, The reinforcement learning motion control framework includes: Define the state variable S of the hexapod robot in the simulation environment, wherein the state variable S includes the pitch angle of the robot platform. Roll angle Linear velocity v and angular velocity of the machine platform , joint angles ; Define the action variable A for a CPG gait generator based on an improved Hopf oscillator, the action variable including a footprint coefficient. Each oscillator i and j The phase difference between ; Construct a control network for a hexapod robot, the control network comprising a policy network, a total state value function network, and two action networks. State-value function network; each network structure is a neural network structure, including an input layer, hidden layers and an output layer.

4. The reinforcement learning-based fault-tolerant gait planning method for a hexapod robot according to claim 3, characterized in that, The reward function of the policy network includes forward distance, body roll degree, energy efficiency, and joint angle mutation, and its expression is: In the formula, r is the reward function. The pitch angle, For roll angle, For the time difference, d Indicates the direction the robot is moving. Let be the distance traveled from time t-1 to time t. For joint torque, For joint velocity, This represents the abrupt change in joint t-1 to time t. , , , All coefficients are user-defined.

5. The reinforcement learning-based fault-tolerant gait planning method for a hexapod robot according to claim 3, characterized in that, The policy network comprises a four-layer neural network structure: an input layer, two hidden layers, and an output layer, which maps the state space to the action space.

6. The reinforcement learning-based fault-tolerant gait planning method for a hexapod robot according to claim 3, characterized in that, The total state value function network includes an online state value function network and a target state value function network, which have the same structure.

7. The reinforcement learning-based fault-tolerant gait planning method for a hexapod robot according to claim 3, characterized in that, The action State-value function networks map state space to actions. A network with a state-value space, two actions The structures of state-value function networks are the same.

8. The reinforcement learning-based fault-tolerant gait planning method for a hexapod robot according to claim 3, characterized in that, The action The parameter updates of the state-value function network use MSELoss as the loss function.

Images