Snake robot swing motion gait generation method and system based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's snake robot swing motion gait generation method and system, it is related to snake robot movement gait design technical field.The technical points of the present application include: establishing snake robot swing motion simulation environment, and optimal joint parameter is obtained based on optimal excitation trajectory calculation, wherein multiple joints of the snake robot are orthogonally arranged alternately, each joint contains multiple joint parameters;Optimal joint parameter is used as the environment of reinforcement learning, and the strategy of reinforcement learning is trained and obtained, and the strategy is the swing torque size of each joint of the snake robot;The swing torque direction of each joint of the snake robot is designed based on energy pumping and collision rules;According to the swing torque size and direction of each joint of the snake robot, swing motion gait is generated.The application effectively solves the problem of torque saturation, and provides a basis for the large-scale cross-bar movement of the snake robot between tree trunk and branch, branch and branch, and tree and tree.

Description

Technical Field

[0001] This invention relates to the field of snake robot gait design technology, and in particular to a method and system for generating swaying gait of a snake robot based on reinforcement learning. Background Technology

[0002] As a type of biomimetic robot, snake-like robots can adapt to various environments by changing their own configuration, exhibiting diverse gaits and postures. They also possess simple orthogonal joint configurations. Among them, three-dimensional snake-like robots without passive wheels are expected to achieve biomimetic functions such as climbing trees, thereby performing tasks such as reconnaissance.

[0003] The ability of snake-like robots to move among trees is fundamental to these reconnaissance tasks. The ability to move between tree trunks and branches, between branches, and between trees allows snake-like robots to better perform reconnaissance missions. And to improve the snake-like robot's ability to move among trees, an efficient tree-movement gait is essential.

[0004] Currently, there are two main categories of methods for designing the gait of snake robots. The first category is the wave function method. [1] The first type involves assigning a trigonometric function to each joint of the snake robot, with the joint number and time as input parameters. Other given parameters vary and are calculated using gait geometry parameters. By substituting the time and joint angle ordinal at each moment, a set of angles that allow the snake robot to exhibit its gait can be calculated. The second type is the curve programming method. [2] The target gait curve is broken down into simple curve segments with known geometric parameters, and the positional relationship between the curve segments is defined. The snake robot is shorter than the target gait curve and is a segment of the target gait curve. Based on the position of the snake robot on the target curve, the current joint angle can be calculated by integration.

[0005] By constructing motion gait based on curve splicing or wave functions, snake-like robots can achieve movements such as climbing tree trunks and straight ladders. However, using static programming methods, namely curve splicing and wave function methods, can only achieve single-pole climbing or small-range pole-crossing movements, and the movement time is long and the speed is slow. For large-scale pole-crossing movements between tree trunks and branches, branches and branches, and trees, static programming methods struggle to overcome the torque saturation problem.

[0006] Therefore, the question of how to design the tree-movement gait of snake robots to improve their movement performance urgently needs to be addressed. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method and system for generating gait patterns of snake-like robots based on reinforcement learning.

[0008] According to one aspect of the present invention, a method for generating the swaying gait of a snake-like robot based on reinforcement learning is proposed, the method comprising:

[0009] A simulation environment for the oscillating motion of a snake-like robot is established, and the optimal joint parameters are calculated based on the optimal excitation trajectory; wherein, the multiple joints of the snake-like robot are arranged orthogonally alternately, and each joint contains multiple joint parameters;

[0010] The optimal joint parameters are used as the environment for reinforcement learning to train and obtain a reinforcement learning strategy, wherein the strategy is the magnitude of the swing torque of each joint of the snake robot.

[0011] The direction of the swing torque of each joint of the snake robot is designed based on the rules of energy pumping and collision.

[0012] The oscillating gait is generated based on the magnitude and direction of the swinging torque of each joint of the snake robot.

[0013] Furthermore, each joint includes the following joint parameters: joint mass, joint coordinates, principal moment of inertia, joint coordinate transformation vector, armature inertia, and coefficient of friction.

[0014] Furthermore, the step of calculating and obtaining the optimal joint parameters based on the optimal excitation trajectory includes: determining the optimal excitation trajectory; in a real environment, making the snake robot move along the optimal excitation trajectory to obtain the actual values ​​of the swing torque of each joint of the snake robot, determining the objective function as the difference between the actual value and the theoretical value of the swing torque, and determining the constraint conditions; and using simulation software to optimize and solve the objective function to obtain the optimal joint parameters.

[0015] Further, determining the optimal excitation trajectory includes: establishing a snake-like robot dynamic model T; linearizing the dynamic model, expressed as: T = W × P, where W represents a matrix composed of joint angles, angular velocities, and angular accelerations, and P represents all joint parameters; and performing QR decomposition on the linearized dynamic model to obtain the maximum linearly independent matrix W. min and the minimum parameter set P min The linearized dynamic model is transformed into: T′=W min ×P min Solve for multiple sets of excitation trajectories to obtain the maximum linearly independent matrix W. min The excitation trajectory with the smallest condition number is the optimal excitation trajectory.

[0016] Furthermore, the dynamic model T consists of multiple T i Composition, T i Represented as:

[0017]

[0018] Where, n iz The torque n at the i-th joint is represented by i Z-axis component; I ai ddQZ represents the armature inertia of the i-th joint; i f represents the coordinates of the angular acceleration of the i-th joint angle in the joint coordinate system; vi f ci dQZ represent the coefficients of friction, respectively. i represents the coordinates of the angular velocity of the i-th joint angle in the joint coordinate system; sgn represents the sign function.

[0019] Furthermore, the process of solving for multiple sets of excitation trajectories results in the maximum linearly independent matrix W. min The optimal excitation trajectory is the one with the minimum condition number. This includes multiple sets of excitation trajectories composed of first- to fifth-order sine curves and first- to fifth-order cosine curves. Joint angles, angular velocities, and angular accelerations are obtained by sampling at the actual sampling frequency, and then substituted to obtain the maximum linearly independent matrix W. min The optimal excitation trajectory is obtained by optimizing the condition number using simulation software.

[0020] Furthermore, the objective function corresponding to the difference between the actual and theoretical values ​​of the swing torque is expressed as:

[0021] min((ww×P min ′-T″) 2 );

[0022] In the formula, min represents minimization; ww represents the maximum linearly independent matrix corresponding to the actual value of the oscillating torque; T″ represents the theoretical value of the oscillating torque composed of multiple joint torques without considering friction and armature inertia; P min ′ represents the minimum set of parameters corresponding to the actual value of the oscillating torque;

[0023] The constraint condition is set as follows: joint mass m i The error between the actual weighing and the pre-set threshold is not greater than the pre-set threshold; the centroid coordinates of the joint are m. i1 m i2 m i3 The length, width, and height of the joint shall not exceed the length, width, and height of the joint itself; the principal moment of inertia of the joint I i11 I i22 I i33 The sum of any two terms is greater than the third term; the sum of the squares of the sine and cosine values ​​of the fixed angles α1, β1, and γ1 in the joint coordinate transformation vector is 1, and the sine and cosine values ​​are between ±1; the armature inertia I of the joint. ai friction coefficient f vi f ci All are greater than 0.

[0024] Furthermore, the reinforcement learning strategy includes a swing motion expert strategy in a no-delay environment and a swing motion expert strategy in a delayed environment, wherein the swing motion expert strategy in a no-delay environment corresponds to the first reinforcement learning model, and the swing motion expert strategy in a delayed environment corresponds to the second behavior cloning model.

[0025] Using the optimal joint parameters as the environment for reinforcement learning, the training yields reinforcement learning strategies including:

[0026] During sampling, the state-action pair at the current time step is used to obtain the state at the next time step. The state at the next time step is then used as the input to the first reinforcement learning model to generate the action 'a' at the next time step. t+1 Simultaneously, the current state-action pair is used as input, and the policy network of the second-action cloning model is used to generate the action a′ for the next time step. t+1 Action a′ t+1 As the action input for the next moment, interact with the environment; repeat the above sampling process to obtain sequence data, form a trajectory, and put the obtained sequence data into the experience playback pool;

[0027] During the update, N samples are randomly selected from the experience replay pool, and the action 'a' generated by the first reinforcement learning model is used as the basis for the update. t+1 As a teacher network, the action a′ generated by the policy network of the second-behavior cloning model. t+1 As a student network, the loss function is calculated as follows:

[0028]

[0029] Next, backpropagation is performed to calculate the gradient and update the weights of the policy network.

[0030] Furthermore, the design rules for the direction of the swing torque of each joint of the snake-like robot include:

[0031] In the snake-like robot oscillating motion simulation environment, the snake-like robot is wrapped around a horizontal bar, establishing a reference axis l1 parallel to the horizontal bar and a reference axis l2 perpendicular to the ground and pointing upwards; the reference axis l1 is connected to the joint axis l j Let α be the included angle between the reference axis l2 and the joint axis l. j The included angle is denoted as β. Based on the relationship between α and β, the j-th joint and the joints following it are determined to be pitch joints or yaw joints.

[0032] The direction of the torque at the pitch joint is the same as the angular velocity of the first joint angle;

[0033] When the angular velocity of the first joint is positive and it swings to the right, the yaw joint moves to the left; when the angular velocity of the first joint is negative and it swings to the left, the yaw joint moves to the right.

[0034] According to another aspect of the present invention, a serpentine robot gait generation system based on reinforcement learning is proposed, the system comprising:

[0035] The environmental data acquisition module is configured to establish a simulation environment for the swaying motion of the snake robot and calculate the optimal joint parameters based on the optimal excitation trajectory; wherein, the multiple joints of the snake robot are arranged orthogonally alternately, and each joint contains multiple joint parameters;

[0036] The strategy training module is configured to use the optimal joint parameters as the environment for reinforcement learning, and to train a reinforcement learning strategy, wherein the strategy is the magnitude of the swing torque of each joint of the snake robot.

[0037] The orientation design module is configured to design the direction of the swing torque of each joint of the snake robot based on energy pumping and collision rules.

[0038] The gait generation module is configured to generate oscillating gait based on the magnitude and direction of the swing torque of each joint of the snake robot.

[0039] The present invention has the following technical effects:

[0040] This invention designs an efficient swinging gait based on reinforcement learning for the inter-tree movement of snake-like robots. Compared with existing gaits designed using static programming methods such as curve stitching and wave functions, this invention effectively solves the torque saturation problem and provides a foundation for snake-like robots to perform large-scale cross-branch movements between tree trunks and branches, branches and branches, and between trees. Furthermore, the swinging gait designed in this invention can be applied to other high-redundancy robots controlled by torque to perform other functions. Attached Figure Description

[0041] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a method for generating the swaying motion gait of a snake-like robot based on reinforcement learning, as described in an embodiment of the present invention.

[0043] Figure 2 This is a front view of the snake-shaped robot swing test frame in an embodiment of the present invention;

[0044] Figure 3 This is a right view of the snake-shaped robot swing test frame in an embodiment of the present invention;

[0045] Figure 4 This is a top view of the snake-shaped robot swinging test frame in an embodiment of the present invention;

[0046] Figure 5 This is a diagram of the algorithm framework in a latency-free environment according to an embodiment of the present invention;

[0047] Figure 6 This is a diagram of the algorithm framework under a delayed environment in an embodiment of the present invention;

[0048] Figure 7 Schematic diagram of the experimental platform in this embodiment of the invention;

[0049] Figure 8 This is an experimental effect diagram from an embodiment of the present invention;

[0050] Figure 9 This is a schematic diagram of a snake robot gait generation system based on reinforcement learning, as described in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0052] This invention proposes a reinforcement learning-based method for generating the swaying gait of a snake-like robot, such as... Figure 1 As shown, the method includes the following steps:

[0053] S1. Establish a simulation environment for the oscillating motion of a snake robot, and calculate and obtain the optimal joint parameters based on the optimal excitation trajectory; wherein, the multiple joints of the snake robot are arranged orthogonally alternately, and each joint contains multiple joint parameters; the joint parameters contained in each joint include: joint mass, joint coordinates, principal rotational inertia of the joint, joint coordinate transformation vector, armature inertia and friction coefficient;

[0054] S2. Using the optimal joint parameters as the environment for reinforcement learning, a reinforcement learning strategy is trained to obtain the strategy, which is the magnitude of the swing torque of each joint of the snake robot.

[0055] S3. Design the direction of the swing torque of each joint of the snake robot based on energy pumping and collision rules;

[0056] S4. Generate swaying gait based on the magnitude and direction of the swinging torque of each joint of the snake robot.

[0057] The method begins with S1. In S1, a simulation environment for the oscillating motion of a snake-like robot is established, and the optimal joint parameters are calculated based on the optimal excitation trajectory.

[0058] According to an embodiment of the present invention, firstly, a simulation environment for the swaying motion of a snake-like robot is built based on the identification of a semi-positive definite programming system.

[0059] Figures 2-4 A schematic diagram of a snake-like robot swinging experimental setup is shown. The blue curve represents the fixed base of the snake-like robot, which remains stationary; the white circle represents a joint of the swinging section of the snake-like robot; the black dashed line represents the reference axis l1, parallel to the horizontal bar and pointing outwards perpendicular to the paper; the red dashed line represents the joint axis l. j The green dashed line represents the reference axis l2 perpendicular to the ground, and α represents l1 and l2. j The included angle, β is l2, l j The included angle; j1, j2, j3, j4... represent joints 1, 2, 3, 4... of the swinging part of the snake robot...

[0060] The snake-like robot's swinging section has 8 joints, each with 16 joint parameters: m i m i1 m i2 m i3 I i11 I i22 I i33 , sinα1, sinβ1, sinγ1, cosα1, cosβ1, cosγ1, I ai f vi f ci Among them, m i Let m be the mass of the i-th joint. i1 m i2 m i3 Let I be the coordinate of the i-th joint in the link coordinate system. i11 I i22 I i33 Let be the principal rotational inertia of the i-th joint module; the joint coordinate transformation vector includes sinα1, sinβ1, sinγ1, cosα1, cosβ1, cosγ1, α1, β1, γ1, which are the fixed XYZ angles of the rotation from the joint coordinate system to the principal axis coordinate system of the i-th joint module, and sinα1, sinβ1, sinγ1, cosα1, cosβ1, cosγ1, and cosγ1 are their respective sine and cosine values; I ai f represents the armature inertia. vi f ci This represents the coefficient of friction.

[0061] Then, perform the following steps: S11, determine the optimal excitation trajectory; S12, in the actual environment, make the snake robot move according to the optimal excitation trajectory, obtain the actual value of the swing torque of each joint of the snake robot, determine the objective function as: the difference between the actual value and the theoretical value of the swing torque, and determine the constraint conditions; S13, use simulation software to optimize and solve the objective function to obtain the optimal joint parameters.

[0062] Specifically, determining the optimal excitation trajectory in S11 includes: S111, establishing a snake-like robot dynamic model T; S112, linearizing the dynamic model, expressed as: T = W × P, where W represents a matrix composed of joint angles, angular velocities, and angular accelerations, and P represents all joint parameters; S113, performing QR decomposition on the linearized dynamic model to obtain the maximum linearly independent matrix W. min and the minimum parameter set P min The linearized dynamic model is transformed into: T′=W min ×P min S114. Solve for multiple sets of excitation trajectories, and the excitation trajectory that minimizes the condition number of the maximum linearly independent matrix is ​​the optimal excitation trajectory.

[0063] Specifically, in S111, the oscillating part of the snake robot is modeled dynamically. Since the fixed base of the snake robot is stationary, equivalent to the flange of the robotic arm, the oscillating part of the snake robot can be modeled as a robotic arm. The dynamic modeling is performed using the Newton-Euler method as follows.

[0064] The length of each joint link is d, and the joint angle of the i-th joint is denoted as q. i The joint angular velocity of the i-th joint is denoted as dq. i The joint angular acceleration of the i-th joint is denoted as ddq. i The rotation coordinate system from the i-th joint to the (i+1)-th joint is:

[0065] i is an even number;

[0066] i is an odd number;

[0067] Among them, c i+1 s i+1 Let i represent the sine and cosine values ​​of the i-th joint angle, where i = 0, 1, 2, ..., 7. Here, i = 0 represents the world coordinate system, and the others represent the i-th joint coordinate system.

[0068] Next, vectorization is performed:

[0069]

[0070] Among them, dQZi This represents the coordinates of the angular velocity of the i-th joint in the joint coordinate system.

[0071]

[0072] Among them, ddQZ i This represents the coordinates of the angular acceleration of the i-th joint angle in the joint coordinate system.

[0073]

[0074]

[0075] in, This represents the distance between the origin of the world coordinate system and the first joint coordinate system in the world coordinate system. The distance between the origin of the i-th joint coordinate system and the (i+1)-th joint coordinate system is represented by the coordinates in the i-th joint coordinate system.

[0076]

[0077] Among them, P ci This represents the coordinates of the centroid of the i-th joint in the i-th joint coordinate system.

[0078]

[0079] Among them, I comi Let represent the inertia matrix of the i-th joint module in the principal axis coordinate system.

[0080]

[0081] Among them, R comi This represents the rotation matrix that rotates from the origin at the center of mass, with the same orientation as the coordinate system of the i-th joint, to the coordinate system of the principal axis of the i-th joint.

[0082] I ci =R comi ×I comi ×R comi T ;

[0083] Among them, I ci This represents the inertia matrix of the i-th joint module in a coordinate system where the origin is at the centroid and the attitude is the same as the joint coordinate system.

[0084]

[0085]

[0086]

[0087] Where ω0 represents the angular velocity of the world coordinate system, dω0 represents the angular acceleration of the world coordinate system, and dv0 represents the linear acceleration of the world coordinate system.

[0088] Next, the velocity and acceleration are extrapolated:

[0089]

[0090]

[0091]

[0092] dv ci+1 =dω i+1 ×P ci+1 +ω i+1 ×(ω i+1 ×P ci+1 )+dv i+1 ;

[0093] F i+1 =m i+1 ×dv ci+1 ;

[0094] N i+1 =I ci+1 ×dω i+1 +ω i+1 ×(I ci+1 ×ω i+1 );

[0095] Where i = 0, 1, 2, ..., 7. ω i+1 dω represents the angular velocity of the (i+1)th joint coordinate system. i+1 dv represents the angular acceleration of the (i+1)th joint coordinate system. i+1 dv represents the linear acceleration of the (i+1)th joint coordinate system. ci+1 F represents the linear acceleration of the centroid of the (i+1)th joint module. i+1 This indicates that dv is generated at the (i+1)th joint centroid. ci+1 The force required for linear acceleration. N i+1 This indicates that dω is generated at the centroid of the (i+1)th joint. i+1 The torque required to achieve angular acceleration.

[0096] Next, the joint torque is calculated by internal calculation:

[0097] f i =f i+1 +F i ;

[0098]

[0099]

[0100] Among them, i=1, 2, ..., 7, 8, f9=0, n9=0, F9=0, f9=0, f i Let n represent the net force required at the i-th joint. i This represents the torque required for the i-th joint without considering friction and armature inertia, and includes three components along the X, Y, and Z axes: n ix n iy n iz .

[0101] Finally, by adding armature inertia and overcoming friction, we obtain the final joint torque required at the i-th joint:

[0102]

[0103] Among them, T i n represents the joint torque at the i-th joint; iz This represents the torque n at the i-th joint when friction and armature inertia are not considered. i Z-axis component; I ai ddQZ represents the armature inertia of the i-th joint; i f represents the coordinates of the angular acceleration of the i-th joint angle in the joint coordinate system; vi f ci dQZ represent the coefficients of friction, respectively. i represents the coordinates of the angular velocity of the i-th joint angle in the joint coordinate system; sgn represents the sign function. i = 1, 2, ..., 7, 8.

[0104] In S112, the dynamic model is linearized and expressed as: T = W × P.

[0105] Specifically, let P be the joint parameters of all joints in the swinging part of the snake robot, which is a 128×1 vector. The goal of linearization is to find a matrix W such that the following equation holds:

[0106] T = W × P;

[0107] Where T represents the joint torque, which is an 8×1 vector, and W is an 8×128 matrix, which is determined solely by the joint angle q. i angular velocity dq i Angular acceleration ddq i composition.

[0108] For the matrix value W[j, k] to be obtained, set the k-th parameter of the joint in the dynamic equation to 1 and set all other parameters to 0, thus obtaining the result determined solely by q. i dq iddq i The matrix value W[j, k] is formed.

[0109] In S113, the linearized dynamic model is decomposed using QR decomposition to obtain the maximum linearly independent matrix W. min and the minimum parameter set P min The linearized dynamic model is transformed into: T′=W min ×P min .

[0110] Specifically, for a matrix W, its column vectors are not linearly independent. Therefore, we need to find the maximal linearly independent set of its column vectors to form matrix W. min Its corresponding parameter set is the minimum parameter set P. min , so that:

[0111] T′=W min ×P min ;

[0112] Find the minimum parameter set using QR decomposition: randomly generate joint angle, angular velocity, and angular acceleration q. i dq i ddq i Substituting the values ​​into matrix W, we obtain matrix R by QR decomposition. If R[k, k] is not equal to 0, then the k-th parameter belongs to the minimum parameter set, and the k-th column of matrix W belongs to the matrix W. min Otherwise, if the k-th parameter does not belong to the minimum parameter set, the k-th column of matrix W does not belong to matrix W. min Therefore, matrix W is obtained. min Its corresponding parameter set is the minimum parameter set P. min For example, W is an 8×128 matrix, and the largest linearly independent matrix of its column vectors is the matrix from column 1 to column 100, corresponding to P. min That is, lines 1 to 100 of P.

[0113] In S114, multiple sets of excitation trajectories are solved, and the excitation trajectory that minimizes the condition number of the maximum linearly independent matrix is ​​the optimal excitation trajectory.

[0114] Specifically, it is assumed that the multiple excitation trajectories of the oscillating part of the snake robot consist of first to fifth order sine curves and first to fifth order cosine curves; the joint angles, angular velocities, and angular accelerations q are obtained by sampling at the actual sampling frequency. i dq i ddq i Substituting into the matrix, we get matrix W. min All W min The trajectories are assembled into a matrix ww in the row direction; the multiple excitation trajectories are optimized to produce a matrix W. minMinimize the condition number. As an example, optimize matrix W using MATLAB's fmincon() function. min The optimal excitation trajectory is obtained by calculating the condition number.

[0115] Then in S12, the snake robot is made to move along the optimal excitation trajectory in the actual environment, and the actual values ​​of the swing torque of each joint of the snake robot are obtained. The objective function is determined as the difference between the actual value and the theoretical value of the swing torque, and the constraint conditions are determined.

[0116] Specifically, firstly, the optimal excitation trajectory is sent as a position command to the snake-like robot in the actual environment. The angle, angular velocity, and current of the actual snake-like robot are read. The angular velocity is differentially calculated to obtain the angular acceleration. Based on the current-torque performance curve of the servo motor, the current is converted into torque. Then, the obtained angle, angular velocity, angular acceleration, and torque are filtered. As an example, the filter is selected using the `butter()` function in MATLAB. Next, the parameter identification of the robot arm with constraints is transformed into a semi-positive definite programming problem. As an example, optimization can be performed using the YAMLIP toolbox.

[0117] The filtered angle, angular velocity, and angular acceleration q i dq i ddq i Substituting, we obtain matrix W. min All W min By assembling these matrices into a matrix ww along the row direction, we obtain the equation for the actual torque value: T 实 =ww×P min The objective function is to minimize the difference between the actual and theoretical values ​​of the oscillating torque, expressed as:

[0118] min((ww×P min ′-T″) 2 );

[0119] In the formula, min represents minimization; ww represents the maximum linearly independent matrix corresponding to the actual value of the oscillating torque; T″ represents the theoretical value of the oscillating torque composed of multiple joint torques without considering friction and armature inertia; P min ′ represents the minimum set of parameters corresponding to the actual value of the oscillating torque;

[0120] The constraint condition is set as follows: joint mass m i The error between the actual weighing and the actual weight does not exceed a preset threshold (e.g., 0.1 kg); the centroid coordinates of the joint are m i1 m i2 m i3 The length, width, and height of the joint shall not exceed the length, width, and height of the joint itself; the principal moment of inertia of the joint I i11 I i22 I i33The sum of any two terms is greater than the third term; the sum of the squares of the sine and cosine values ​​of the fixed angles α1, β1, and γ1 in the joint coordinate transformation vector is 1, and the sine and cosine values ​​are between ±1; the armature inertia I of the joint. ai friction coefficient f vi f ci All are greater than 0.

[0121] Then, in S13, the objective function is optimized and solved using simulation software to obtain the optimal joint parameters. As an example, the solvesdp() function from the YAMLIP toolbox is used for optimization, and the minimum set of parameters obtained is the optimal joint parameters.

[0122] Furthermore, the optimal joint parameters are transformed into a complete parameter set by using the matrix R obtained from QR decomposition. The complete parameter set is then matched one-to-one with the parameters in the files of the open-source simulation environment to obtain a simulation environment that conforms to reality.

[0123] Then, after calculating and obtaining the optimal joint parameters based on the optimal excitation trajectory, step S2 is executed. In step S2, the optimal joint parameters are used as the environment for reinforcement learning to train and obtain a reinforcement learning policy, which is the magnitude of the swing torque of each joint of the snake robot; wherein, the reinforcement learning policy includes a swing motion expert policy in a delay-free environment and a swing motion expert policy in a delay environment, the swing motion expert policy in a delay-free environment corresponds to the first reinforcement learning model, and the swing motion expert policy in a delay environment corresponds to the second behavior cloning model.

[0124] According to an embodiment of the present invention, the first reinforcement learning model corresponding to the expert policy of oscillating motion in a latency-free environment can adopt an existing model, such as... Figure 5 As shown, as an example, the first reinforcement learning model structure includes an action value function network, a hidden state encoding network, a dynamics model network, a reward network, and a policy network. The input of the action value function network is the hidden state and the action, and the output is the state value function; the input of the hidden state encoding network is the state, and the output is the encoded hidden state; the input of the dynamics model network is the hidden state and the action, and the output is the hidden state at the next time step; the input of the reward network is the hidden state and the action, and the output is the reward at the next time step; the input of the policy network is the state, and the output is the action; the reward is set as the mechanical energy of the swaying part of the snake robot.

[0125] The loss functions for hidden state coding networks, dynamic model networks, reward networks, and action-value function networks are shown below:

[0126]

[0127]

[0128]

[0129] loss=consistent_loss+reward_loss+value-loss;

[0130] Where N represents the number of samples drawn from the experience replay pool; horizon represents the horizon of the model predictive control; z i+t Represents the state at time i+t; z i+tDred r represents the state prediction at time i+t; i+t r represents the reward at time i+t; i+tDred Represents the predicted reward at time i+t; represents the output of the action-value function network at time i+t; (Q(z) i+t a i+t )-(r+Q(z i+t+1 a i+t+1 ))) 2 td represents the error of the action value function network at time i+t; rho represents a constant factor that attenuates the error at later time steps.

[0131] The policy network is updated according to the following formula:

[0132]

[0133] Among them, Q(z) i+t a i+t ) represents the output of the action value function network at time i+t, and rho represents a constant factor that attenuates the output of the action value network at subsequent time steps.

[0134] Each time the model samples one or more trajectories to reach a set amount of data, the weights of the hidden state encoding network, dynamics model network, reward network, action value function network, and policy network are updated once. Each weight update includes the following two steps:

[0135] 1) Sampling: First, the current state s is sampled using a hidden state coding network. t Encoded as hidden state z t The current policy network π is based on the hidden state z t The mean μ of the Gaussian strategy is obtained, and a series of actions a are obtained by sampling using a Gaussian distribution. π Next, the loop continues, using the actions predicted at time t-1 and thereafter as the mean. If no mean exists, the mean is set to 0. A series of actions a are obtained by sampling using a Gaussian distribution. g The action a is analyzed using a dynamic model network, a reward network, and an action value function network. g and action a π An evaluation is conducted, and the action a with the highest cumulative return after a certain number of discounts is selected.elite Generate the mean and variance for the next iteration to generate a. g After the loop completes, the optimal sequence action 'a' is generated, which is a sequence action 'a' of length 'horizon'. t a t+1 ,..,a t+horizon . a t As input, it interacts with the environment to obtain the state s for the next time step. t+1 Receive immediate reward at this moment. t Then, based on the state s at the next time step t+1 Repeat the previous action selection process to obtain the next action a. t+1 Repeat this sampling process to obtain sequence data, which forms a trajectory. Where T is the termination time. The obtained sequence data is placed into the experience replay pool.

[0136] 2) Update: Randomly select N samples from the experience replay pool, and calculate the loss and L using the loss function described above. π Then, backpropagation is performed to calculate the gradient and update the weights of the hidden state encoding network, dynamic model network, reward network, action value function network, and policy network.

[0137] However, in real-world environments, there are latency issues related to communication and computation. Therefore, this invention further designs a second behavior cloning model to train expert strategies for oscillating motion in environments with latency.

[0138] like Figure 6 As shown, in a simulation environment with delay, the system uses the current state action pair (s) to determine the time interval. t a t The action a at the next moment is calculated. t+1 As the action input for the next moment, the state-action pair (s) for the next moment is obtained. t+1 a t+1 Repeating the above process yields a trajectory. The second line of the cloning model structure includes a policy network, whose input is the current state-action pair (s). t a t The output is the action 'a' at the next moment. t+1 Each time the model samples, the weights of the policy network are updated once. Each weight update includes the following two steps:

[0139] 1) Sampling: Using the current state and action pair (s) t a t ) Obtain the state s at the next moment. t+1 Use the state s of the next moment. t+1The input to the first reinforcement learning model generates the action 'a' for the next time step. t+1 At the same time, the current state action pair (s) is... t a t Using the second line as input, the policy network of the clone model generates the action a′ for the next time step. t+1 Action a′ t+1 This serves as the action input for the next moment and the interaction with the environment. This sampling process is repeated to obtain sequence data, forming a trajectory, which is then placed into the experience playback pool.

[0140] 2) Update: Randomly select N samples from the experience replay pool, and the action a generated by the first reinforcement learning model. t+1 As a teacher network, the action a′ generated by the policy network of the second-behavior cloning model. t+1 As a student network, the loss function is calculated as follows:

[0141]

[0142] After calculating the loss, backpropagation is then performed to calculate the gradient and update the weights of the policy network.

[0143] Then, S3 is executed. In S3, the swing torque direction of each joint of the snake robot is designed based on energy pumping and collision rules. This includes: in the snake robot swing motion simulation environment, the snake robot is wrapped around a horizontal bar, and a reference axis l1 parallel to the horizontal bar and a reference axis l2 perpendicular to the ground are established; the reference axis l1 is then connected to the joint axis l2. j Let α be the included angle between the reference axis l2 and the joint axis l. j The included angle is denoted as β. Based on the relationship between α and β, the j-th joint and the joints following it are determined to be either pitch joints or yaw joints. The torque direction of the pitch joint is consistent with the angular velocity of the first joint angle. When the angular velocity of the first joint angle is positive and it swings to the right, the yaw joint moves to the left. When the angular velocity of the first joint angle is negative and it swings to the left, the yaw joint moves to the right.

[0144] According to an embodiment of the present invention, a rule for setting the torque direction of joint modules is proposed, enabling a snake-like robot to move in a swaying gait. Adjacent joints of the snake-like robot are arranged orthogonally. First, for the first joint j1 of the swaying section of the snake-like robot, if α > β, then this joint is designated as the pitch joint, and subsequent joints j2, j3, j4 are designated as the yaw joint, pitch joint, yaw joint, and so on. If α < β, then this joint is designated as the yaw joint, and subsequent joints j2, j3, j4 are designated as the pitch joint, yaw joint, pitch joint, and so on. Next, based on the rule of pumping energy, the torque direction of the pitch joint must be consistent with the angular velocity of the first joint, pumping energy to increase the total mechanical energy of the system. Secondly, based on the principle of avoiding collisions with the base, the torque direction of the yaw joint is further determined. Figure 2 As shown, when the angular velocity of the first joint of the snake robot is positive and it swings to the right, it needs to avoid colliding with the base. Therefore, the yaw joint should move to the left. When the angular velocity of the first joint of the snake robot is negative and it swings to the left, the yaw joint should move to the right. The first joint can be determined by the order of the joints arranged from top to bottom.

[0145] Finally, S4 is executed, in which the oscillating gait is generated based on the magnitude and direction of the swinging torque of each joint of the snake robot.

[0146] The technical effects of the present invention were further verified through experiments.

[0147] like Figure 7 As shown, the constructed snake-like robot has 26 joints, a total length of 1.9m, a diameter of 0.066m, and weighs approximately 7kg. The servo motors used are Dynamixel XH540-W270s, divided into two groups. The servo motors in the same group have parallel axes facing the same direction, while adjacent servo motor axes are perpendicular. Custom-designed connectors are used between the servo motors, and a comb-shaped nylon shell is installed externally for protection. Starting from the head, 19 modules are wound around a horizontal bar and fixed by fasteners, forming the base. The remaining 8 joints are suspended in mid-air, forming the swinging section. After generating a swinging strategy under a delayed environment, communication is established between the PC and the snake-like robot. Power is supplied via a power source, and the servo angles and angular velocities are read using a 485 serial port. These values ​​are input into the swinging strategy to calculate joint torques in real time and then sent to the robot.

[0148] Experimental results are as follows Figure 8 As shown, Figure 8 This demonstrates the oscillating motion capability of the snake-like robot in the three-dimensional gait designed in this invention. Each small image is spaced 0.2 seconds apart. Figure 8 As can be seen from this, the gait of the present invention mimics swing motion, and can accumulate energy under relatively small torque constraints to achieve a high swing height and swing speed.

[0149] This invention proposes a method for generating gait of a snake robot based on reinforcement learning. Compared with existing static planning methods such as curve splicing and wave function, this method can effectively solve the problem of torque saturation and provides a foundation for the snake robot to perform large-scale pole-crossing movements between tree trunks and branches, branches and branches, and trees. Therefore, it has high practical application value.

[0150] Another embodiment of the present invention proposes a gait generation system for snake-like robots based on reinforcement learning, such as... Figure 9 As shown, the system includes:

[0151] The environmental data acquisition module 910 is configured to establish a simulation environment for the swaying motion of the snake robot and calculate and obtain the optimal joint parameters based on the optimal excitation trajectory. The multiple joints of the snake robot are arranged orthogonally and alternately, and each joint contains multiple joint parameters. The joint parameters contained in each joint include: joint mass, joint coordinates, principal moment of inertia of the joint, joint coordinate transformation vector, armature inertia, and friction coefficient.

[0152] The strategy training module 920 is configured to use the optimal joint parameters as the environment for reinforcement learning to train and obtain a reinforcement learning strategy, wherein the strategy is the magnitude of the swing torque of each joint of the snake robot.

[0153] The orientation design module 930 is configured to design the direction of the swing torque of each joint of the snake robot based on energy pumping and collision rules.

[0154] The gait generation module 940 is configured to generate a swaying gait based on the magnitude and direction of the swinging torque of each joint of the snake robot.

[0155] In this embodiment, preferably, the calculation of optimal joint parameters based on the optimal excitation trajectory in the environmental data acquisition module 910 includes: determining the optimal excitation trajectory; in the actual environment, making the snake robot move according to the optimal excitation trajectory, obtaining the actual value of the swing torque of each joint of the snake robot, determining the objective function as: the difference between the actual value and the theoretical value of the swing torque, and determining the constraint conditions; using simulation software to optimize and solve the objective function to obtain the optimal joint parameters.

[0156] In this embodiment, preferably, the determination of the optimal excitation trajectory in the environmental data acquisition module 910 includes: establishing a snake-like robot dynamic model T; linearizing the dynamic model, representing it as: T = W × P, where W represents a matrix composed of joint angles, angular velocities, and angular accelerations, and P represents all joint parameters; performing QR decomposition on the linearized dynamic model to obtain the maximum linearly independent matrix W. min and the minimum parameter set P minThe linearized dynamic model is transformed into: T′=W min ×P min Solve for multiple sets of excitation trajectories to obtain the maximum linearly independent matrix W. min The excitation trajectory with the smallest condition number is the optimal excitation trajectory.

[0157] In this embodiment, preferably, the objective function corresponding to the difference between the actual value and the theoretical value of the swing torque in the environmental data acquisition module 910 is expressed as:

[0158] min((ww×P min ′-T″) 2 );

[0159] In the formula, min represents minimization; ww represents the maximum linearly independent matrix corresponding to the actual value of the oscillating torque; T″ represents the theoretical value of the oscillating torque composed of multiple joint torques without considering friction and armature inertia; P min ′ represents the minimum set of parameters corresponding to the actual value of the oscillating torque;

[0160] The constraint condition is set as follows: joint mass m i The error between the actual weighing and the pre-set threshold is not greater than the pre-set threshold; the centroid coordinates of the joint are m. i1 m i2 m i3 The length, width, and height of the joint shall not exceed the length, width, and height of the joint itself; the principal moment of inertia of the joint I i11 I i22 I i33 The sum of any two terms is greater than the third term; the sum of the squares of the sine and cosine values ​​of the fixed angles α1, β1, and γ1 in the joint coordinate transformation vector is 1, and the sine and cosine values ​​are between ±1; the armature inertia I of the joint. ai friction coefficient f vi f ci All are greater than 0.

[0161] In this embodiment, preferably, the reinforcement learning strategies in the policy training module 920 include swing motion expert strategies in a no-delay environment and swing motion expert strategies in a delayed environment, wherein the swing motion expert strategy in the no-delay environment corresponds to a first reinforcement learning model, and the swing motion expert strategy in the delayed environment corresponds to a second behavior cloning model; using the optimal joint parameters as the reinforcement learning environment, the reinforcement learning strategies obtained by training include:

[0162] During sampling, the state-action pair at the current time step is used to obtain the state at the next time step. The state at the next time step is then used as the input to the first reinforcement learning model to generate the action 'a' at the next time step. t+1 Simultaneously, the current state-action pair is used as input, and the policy network of the second-action cloning model is used to generate the action a′ for the next time step.t+1 Action a′ t+1 As the action input for the next moment, interact with the environment; repeat the above sampling process to obtain sequence data, form a trajectory, and put the obtained sequence data into the experience playback pool;

[0163] During the update, N samples are randomly selected from the experience replay pool, and the action 'a' generated by the first reinforcement learning model is used as the basis for the update. t+1 As a teacher network, the action a′ generated by the policy network of the second-behavior cloning model. t+1 As a student network, the loss function is calculated as follows:

[0164]

[0165] Next, backpropagation is performed to calculate the gradient and update the weights of the policy network.

[0166] In this embodiment, preferably, the design rules for the swing torque direction of each joint of the snake robot in the direction design module 930 include: in the snake robot swing motion simulation environment, the snake robot is wrapped around a horizontal bar, and a reference axis l1 parallel to the horizontal bar and a reference axis l2 perpendicular to the ground and upward are established; the reference axis l1 and the joint axis l2 are then connected. j Let α be the included angle between the reference axis l2 and the joint axis l. i The included angle is denoted as β. Based on the relationship between α and β, the j-th joint and the joints following it are determined to be either pitch joints or yaw joints. The torque direction of the pitch joint is consistent with the angular velocity of the first joint angle. When the angular velocity of the first joint angle is positive and it swings to the right, the yaw joint moves to the left. When the angular velocity of the first joint angle is negative and it swings to the left, the yaw joint moves to the right.

[0167] For the undescribed portion of the snake robot gait generation system based on reinforcement learning according to an embodiment of the present invention, please refer to the detailed description of the method embodiment above.

[0168] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

[0169] The following documents are cited in this invention:

[0170] [1]M.Tesch,K.Lipkin,I.Brown,R.Hatton,A.Peck,J.Rembisz&H.Choset(2009)Parameterized and Scripted Gaits for Modular Snake Robots,Advanced Robotics,vol.23,no.9,pp.1131-1158,DOI:10.1163 / 156855309X452566.

[0171] [2]T.Takemori,M.Tanaka and F.Matsuno(2018)Gait Design for a SnakeRobot by Connecting Curve Segments and Experimental Demonstration,IEEETransactions on Robotics,vol.34,no.5,pp.1384-1391,doi:10.1109 / TRO.2018.28303。

Claims

1. A method for generating gait of a snake-like robot based on reinforcement learning, characterized in that, include: A simulation environment for the oscillating motion of a snake-like robot is established, and the optimal joint parameters are calculated based on the optimal excitation trajectory; wherein, the multiple joints of the snake-like robot are arranged orthogonally alternately, and each joint contains multiple joint parameters; The optimal joint parameters are used as the environment for reinforcement learning to train and obtain a reinforcement learning strategy, wherein the strategy is the magnitude of the swing torque of each joint of the snake robot. The direction of the swing torque of each joint of the snake robot is designed based on the rules of energy pumping and collision. The oscillating gait is generated based on the magnitude and direction of the swinging torque of each joint of the snake-like robot. The reinforcement learning strategies include a swing motion expert strategy in a no-delay environment and a swing motion expert strategy in a delayed environment, wherein the swing motion expert strategy in a no-delay environment corresponds to the first reinforcement learning model, and the swing motion expert strategy in a delayed environment corresponds to the second behavior cloning model. Using the optimal joint parameters as the environment for reinforcement learning, the training yields reinforcement learning strategies including: During sampling, the state-action pair at the current time step is used to obtain the state at the next time step. The state at the next time step is then used as the input to the first reinforcement learning model to generate the action for the next time step. Simultaneously, the current state-action pair is used as input, and the policy network of the second-action cloning model is used to generate the action for the next time step. ,action As the action input for the next moment, interact with the environment; repeat the above sampling process to obtain sequence data, form a trajectory, and put the obtained sequence data into the experience playback pool; At the time of updating, a sample is randomly drawn from the experience replay pool The action generated by the first reinforcement learning model The action generated by the policy network of the second behavior cloning model as a teacher network The loss function is calculated as a student network The following formula: ； Next, backpropagation is performed to calculate the gradient and update the weights of the policy network. 2.The method of claim 1, wherein, Each joint includes the following parameters: joint mass, joint coordinates, principal moment of inertia, joint coordinate transformation vector, armature inertia, and coefficient of friction. 3.The method of claim 2, wherein, The process of calculating and obtaining optimal joint parameters based on the optimal excitation trajectory includes: determining the optimal excitation trajectory; in a real environment, making the snake robot move along the optimal excitation trajectory, obtaining the actual values ​​of the swing torque of each joint of the snake robot, determining the objective function as the difference between the actual value and the theoretical value of the swing torque, and determining the constraint conditions; and using simulation software to optimize and solve the objective function to obtain the optimal joint parameters.

4. The method for generating gait of a snake-like robot based on reinforcement learning according to claim 3, characterized in that, Determining the optimal excitation trajectory includes: establishing a dynamic model of the snake-like robot. The dynamic model is linearized and expressed as follows: , This represents a matrix composed of joint angles, angular velocities, and angular accelerations. Represent all joint parameters; perform linearization on the dynamic model. Decomposition to find the largest linearly independent matrix and minimum parameter set The linearized dynamic model is then transformed into: Solve for multiple sets of excitation trajectories to obtain the maximum linearly independent matrix. The excitation trajectory with the smallest condition number is the optimal excitation trajectory.

5. The method for generating gait of a snake-like robot based on reinforcement learning according to claim 4, characterized in that, The kinetic model consists of comprises, is represented as: ; in, This indicates that the first armature inertia is not considered when friction and armature inertia are neglected. Joint torque of Axial components; Indicates the first Armature inertia of each joint; Indicates the first The coordinates of the joint angle acceleration in the joint coordinate system; These represent the coefficients of friction; Indicates the first The coordinates of the angular velocity of each joint angle in the joint coordinate system; Represents a symbolic function.

6. The method for generating the swaying gait of a snake-like robot based on reinforcement learning according to claim 5, characterized in that, The solving of the multiple sets of excitation trajectories is such that the excitation trajectory with the minimum condition number of the maximum linear independent matrix is the optimal excitation trajectory, and the multiple sets of excitation trajectories are composed of first to fifth order sinusoidal curves and first to fifth order cosine curves, the joint angle, angular velocity and angular acceleration are sampled at an actual sampling frequency, and substituted into the maximum linear independent matrix to obtain the optimal excitation trajectory by optimizing and solving the condition number by using simulation software.

7. The method for generating gait of a snake-like robot based on reinforcement learning according to claim 3, characterized in that, The objective function corresponding to the difference between the actual value and the theoretical value of the swing torque is expressed as: ； In the formula, Indicates minimization; This represents the maximum linearly independent matrix corresponding to the actual value of the oscillating torque; This represents the theoretical value of the oscillating torque composed of multiple joint torques when friction and armature inertia are not considered. This represents the minimum set of parameters corresponding to the actual value of the oscillating torque. The constraint condition is set as follows: joint mass The error between the actual weighing and the preset threshold is not exceeded; the centroid coordinates of the joint. The length, width, and height of the joint itself shall not exceed the joint's length, width, and height; the principal moment of inertia of the joint. The sum of any two terms is greater than the third term; fixed angles in the joint coordinate transformation vector. The sum of the squares of the sine and cosine values ​​is 1, and the sum of the squares of the sine and cosine values ​​is in the range of 1 / 2. Between; armature inertia of the joint coefficient of friction All are greater than 0.

8. The method for generating gait of a snake-like robot based on reinforcement learning according to claim 1, characterized in that, The design rules for the direction of the swing torque of each joint of the snake-like robot include: In the simulation environment of the snake-like robot's oscillating motion, the snake-like robot is wrapped around a horizontal bar, establishing a reference axis parallel to the horizontal bar. and the reference axis perpendicular to the ground upwards ; set the reference axis With joint axis The included angle is denoted as , make the reference axis With joint axis The included angle is denoted as ,according to and Determine the size relationship of the first The joints and the joints following them are pitch joints or yaw joints. The direction of the torque at the pitch joint is the same as the angular velocity of the first joint angle; When the angular velocity of the first joint is positive and it swings to the right, the yaw joint moves to the left; when the angular velocity of the first joint is negative and it swings to the left, the yaw joint moves to the right.

9. A reinforcement learning-based serpentine robot gait generation system, applicable to any one of the reinforcement learning-based serpentine robot gait generation methods of claims 1-8, characterized in that, include: The environmental data acquisition module is configured to establish a simulation environment for the swaying motion of the snake robot and calculate the optimal joint parameters based on the optimal excitation trajectory; wherein, the multiple joints of the snake robot are arranged orthogonally alternately, and each joint contains multiple joint parameters; The strategy training module is configured to use the optimal joint parameters as the environment for reinforcement learning, and to train a reinforcement learning strategy, wherein the strategy is the magnitude of the swing torque of each joint of the snake robot. The orientation design module is configured to design the direction of the swing torque of each joint of the snake robot based on energy pumping and collision rules. The gait generation module is configured to generate oscillating gait based on the magnitude and direction of the swing torque of each joint of the snake robot.

Images