Closed chain type mobile robot leg design method based on deep reinforcement learning

Abstract

The invention discloses a closed-chain type mobile robot leg design method based on deep reinforcement learning, which comprises the following steps: S1, constructing an open-chain type eight-degree-of-freedom bionic four-footed intelligent body in a reinforcement learning training environment, and carrying out large-scale iterative training until convergence so as to enable the intelligent body to obtain a motion control strategy for simulating natural walking of an animal; s2, analyzing the motion of the intelligent body based on a forward and inverse kinematics principle, and calculating an optimal adaptive foot end track in a walking process; s3, screening a proper leg configuration in a closed kinematic chain atlas of mechanism according to the track characteristics; s4, a reward mechanism in deep reinforcement learning is used for optimizing and developing a closed-chain leg mechanism size optimizer, the foot end track is input into the optimizer after smooth processing, and the optimal size parameter of the closed-chain leg mechanism can be obtained. The method effectively solves the problem that shape features and time sequence information of the foot end track need to be manually planned, and the movement performance and adaptability of a mechanism are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot mechanism, and in particular to a closed-chain mobile robot leg design method. BACKGROUND

[0002] As a combination of bionics and robotics, legged robots have become a hot topic in the field of robotics in recent years. Compared with traditional wheeled and tracked robots, legged robots have stronger adaptability in unstructured environments due to their high degree of freedom of movement and flexible movement trajectory, especially in uneven terrain, narrow space and complex environment. Inspired by the efficient movement mechanism of natural animals such as leopards, deer and canines, the current mainstream research team's legged robots generally adopt open-chain leg structure, such as Boston Dynamics' BigDog quadruped robot, ETH Zurich's ANYmal robot, and GO2 robot of Yushu Technology. However, this kind of open-chain structure has high movement flexibility, low leg stiffness, and the multi-degree-of-freedom scheme also leads to complex control strategy and high manufacturing cost, so the open-chain leg robot is still limited in mass production and application.

[0003] It is worth noting that the skeletal system of natural organisms is essentially a closed-chain structure. Taking the hind limbs of canines as an example, its hip-knee-ankle joint forms a closed-chain mechanism with passive compliance characteristics through the muscle-ligament system. Inspired by this, the research community began to explore single-degree-of-freedom multi-link walking mechanisms with bionic closed-chain characteristics. Compared with open-chain leg robots, closed-chain robots have fewer driving numbers, are usually supported by multiple branch chains at the foot end, have higher rigidity and carrying capacity, and due to good kinematic characteristics, the repeatability of the end effector (foot end) is higher and the movement is more stable. Therefore, the study of such leg walking mechanisms is of great significance for legged robots.

[0004] For closed-chain legged mobile robots, the leg is limited by single-degree-of-freedom driving closed-chain constraints, resulting in fixed foot trajectory shape and single movement mode. This characteristic makes the geometric characteristics of the foot trajectory (such as stride height, touchdown angle) and the dynamic parameters (such as impact acceleration, ground reaction force distribution) directly determine the motion stability, energy efficiency and terrain adaptability of the whole machine. In the mechanism design process of closed-chain legged robots, the traditional trajectory synthesis method usually adopts the reverse engineering mode: first, obtain the experience gait data through biological observation, then construct the geometric trajectory by polynomial fitting or spline interpolation, and finally match the rod length parameters through optimization algorithm.

[0005] Firstly, such a trajectory of the foot end planned by human mostly focuses on the shape of the spatial path, often ignoring the speed continuity in the time dimension; secondly, in order to more completely reproduce the characteristic information and time information contained in the trajectory of the foot end to the closed chain leg mechanism, a more accurate optimization process is indispensable, and the traditional optimization algorithm (such as genetic algorithm, particle swarm algorithm) is often used for size optimization, which often faces the problems of poor convergence and loss of local details. SUMMARY

[0006] The purpose of the present application is to provide a closed-chain mobile robot leg design method based on deep reinforcement learning, which combines deep reinforcement learning technology with mechanism theory, takes bionic movement as the goal, generates a set of adaptive foot end trajectories containing time dimension, and further deepens the integration of deep reinforcement learning and size optimization of closed chain leg mechanism to obtain the optimal size of the mechanism that can adapt to the foot end trajectory.

[0007] To this end, the present application provides a closed-chain mobile robot leg design method based on deep reinforcement learning, comprising: S1, building an open-chain eight-degree-of-freedom bionic quadruped robot agent in the training environment of deep reinforcement learning, and iterating for multiple times until convergence to make it learn the motion control strategy of simulating the natural walking of animals; S2, analyzing the motion of the agent based on the principle of forward and inverse kinematics, and calculating the optimal foot end trajectory in the walking process of the agent; S3, extracting the foot end trajectory feature from the foot end trajectory, and screening the leg configuration in the closed kinematic chain atlas of mechanism theory according to the foot end trajectory feature; S4, inputting the above foot end trajectory after smoothing into the closed chain leg mechanism size optimizer to obtain the optimal size parameters of the leg configuration, wherein the closed chain leg mechanism size optimizer is developed by using the method of optimizing the reward mechanism in deep reinforcement learning.

[0008] Compared with the prior art, the present application has the following beneficial effects:

[0009] The present application provides a novel closed-chain mobile robot leg design method, which utilizes the advantages of deep reinforcement learning in high-dimensional continuous control tasks, trains an eight-degree-of-freedom open-chain quadruped robot agent to achieve bionic walking on the ground, and generates a foot end motion trajectory with good stability and low energy consumption. This trajectory generation method does not need to set the trajectory shape artificially, and the time sequence information is automatically obtained by simulating the walking process of animals, thereby ensuring the speed continuity of the foot end trajectory. In addition, the combination of the closed chain mechanism size optimizer based on deep reinforcement learning further improves the accuracy of the mapping of the foot end trajectory to the actual mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the application and, together with the description, serve to explain the application without imposing undue limitation thereof. In the drawings:

[0011] Figure 1 is a flow chart of the closed-chain mobile robot leg design method of the present application;

[0012] Figure 2 is a schematic diagram of a simplified model of a quadruped robot of the present application;

[0013] Figure 3 is a schematic diagram of the change in reward value and Q value of an embodiment of the present application;

[0014] Figure 4 is a simplified flow chart of a closed-chain leg mechanism of an embodiment of the present application;

[0015] Figure 5 is a schematic diagram of a closed-chain leg mechanism of an embodiment of the present application;

[0016] Figure 6 is a schematic diagram of a comparison of target trajectory and optimized trajectory of an embodiment of the present application;

[0017] Figure 7 is a schematic diagram of the position fitting result in the x-y direction of an embodiment of the present application;

[0018] Figure 8 is a schematic diagram of the acceleration fitting result in the x-y direction of an embodiment of the present application;

[0019] Figure 9 is a closed-chain octopod mobile robot designed according to the method. DETAILED DESCRIPTION

[0020] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0021] Figure 1 is a flow chart of the closed-chain mobile robot leg design method of the present application, as shown in Figure 1 , comprising the following steps S1 to S4.

[0022] S1, in a deep reinforcement learning training environment, a four-legged robot agent with eight degrees of freedom in an open chain is built.

[0023] In this step, based on deep reinforcement learning technology, we design the physical model of quadruped robot agent and the simulation training environment, and design a reasonable state space, action space, reward function and training algorithm for the agent. After several iterations of training, the quadruped agent masters the motion control strategy of natural walking simulation.

[0024] Among them, the simulation training environment is built on the Matlab platform, and the quadruped robot model is made by Simulink, and its simplified model is as shown in Figure 2 .

[0025] The state space of the quadruped robot agent includes the position, velocity, attitude angle, joint angle and torque of the body, etc. 25-dimensional information; the action space contains eight-dimensional information, which corresponds to the output torque of the eight joints of the robot; the selected deep reinforcement learning training algorithm is TD3 algorithm.

[0026] The reward function of the quadruped robot agent is composed of 5 sub-reward functions, and the total reward R is represented as shown in equation (1), wherein is a positive weighting coefficient:

[0027] (1)

[0028] Among them, is the forward reward value of the robot, which is used to ensure that the robot produces forward motion, and is specifically as shown in equation (2). and are the coordinate values of the robot in the forward direction at the current time and the previous time, respectively and represent the motion speed of the robot at the current time and the previous time, respectively, wherein the logarithmic function is used to ensure the boundedness of the speed. Encourage forward distance, the more forward, the higher the reward; Punish the non-smoothness of the forward speed, the smaller, the higher the reward.

[0029] (2)

[0030] is the center of mass stability reward value, which is used to limit the stability of the robot's body, and is specifically as shown in equation (3). and is the defined body float threshold, when the body center of mass fluctuates within the threshold, it is considered that the motion of the robot meets the expectation and is rewarded. Punish the body fluctuation too large, reward the stability of the body, the more stable the walking process, the higher the reward.

[0031] (3)

[0032] The attitude angle reward value is used to measure the changes in the robot's roll angle, pitch angle and yaw angle, as shown in Equation (4). , and These are the changes in the fuselage roll angle, pitch angle, and yaw angle within a time step, respectively. Logarithmic functions are also used to ensure their boundary properties. This function is an auxiliary reward function to ensure the stability of the fuselage. , and These are used to penalize situations where the robot's roll angle, pitch angle, and yaw angle change excessively.

[0033] (4)

[0034] The reward value is used to maintain the robot's exploration strategy, as shown in equation (5). and These are the sampling time and the simulation time, respectively. This reward function provides the robot with a constant reward at each time step. Encourage robots to remain in an exploratory state.

[0035] (5)

[0036] The energy reward value is used to minimize energy consumption during robot movement, as shown in equation (6). Referring to the principle of minimum energy consumption in nature, animals continuously adjust and optimize their actions to minimize energy consumption. For the constructed quadrupedal intelligent agent, a similar effect can be achieved by limiting joint torques; Let be the torque value of joint i in the previous time step. As training progresses, each subsequent movement of the joint consumes less energy than the previous one, until it converges to a certain stable movement strategy. To encourage robots to walk with low energy consumption, the higher the joint torque required for the next step, the higher the reward will be.

[0037] (6)

[0038] In addition, the quadrupedal agent's reinforcement learning training round has a step length of 500. After exceeding 500 steps, the training automatically resets and begins the next round. Other reset conditions are set as follows: 1. Center of mass height is below 0.5m; 2. Any part of the torso is in contact with the ground; 3. Any joint is in contact with the ground; 4. Roll and pitch angle fluctuations are greater than 10°, or yaw angle fluctuations are greater than 15°. Training stops when the average reward reaches 200. During training, the reward value and Q-value change with the number of rounds as follows: Figure 3 As shown.

[0039] S2. Analyze the motion of the agent based on the principles of forward and inverse kinematics, and calculate the optimal foot trajectory during the agent's walking process.

[0040] The joint data of the quadruped robot after training were sampled at a time interval of 0.025s. Then, the foot trajectory was solved using the open-chain two-degree-of-freedom single-leg kinematics. The pose equation of the foot of the open-chain two-degree-of-freedom single-leg is as shown in equation (7):

[0041] (7)

[0042] in This is the distance between the hip and knee joints. This is the distance between the foot and the knee joint. This refers to the hip joint rotation angle; This represents the rotation angle of the knee joint.

[0043] Further analysis of the shape features and temporal information of the foot trajectory revealed that the foot trajectory has a span height of approximately 544.05 mm, a span distance of approximately 128.54 mm, and a time period of 0.9 s. The trajectory consists of 36 discrete points, which are the foot endpoints generated from the sampled joint data.

[0044] S3. Use the extracted foot trajectory features to filter closed-chain leg configurations.

[0045] Using the extracted foot trajectory features and referring to the "creative design method of mechanical devices" proposed by Professor Yan Hongsen, suitable leg configurations were selected from the closed kinematic chain atlas and simplified for design.

[0046] In one embodiment, the selected closed-chain leg configuration is derived from the (8, 10) type in the closed eight-bar kinematic chain diagram. It is a single degree of freedom, and the operation of the entire leg is driven by the rotation of the crank. The main body of the leg mechanism consists of the frame AD, the power crank AB, and two auxiliary rods. And three three-pole clubs Composition, I is also the foot endpoint; Figure 4 This is a simplified flowchart of the leg structure. Figure 5This is represented by a simplified structural diagram of the leg mechanism.

[0047] S4. After smoothing the above foot trajectory, input it into the closed-chain leg mechanism size optimizer to obtain the optimal size parameters of the leg configuration.

[0048] A closed-loop leg mechanism scale optimizer based on deep reinforcement learning was constructed. The input value is the target trajectory, and the output value is the optimal link length parameter of the leg mechanism. The scale optimizer includes the following steps S41-S44.

[0049] S41. Selection and smoothing of target trajectory T.

[0050] In one embodiment, the selection of the target trajectory T is mainly based on the 36 discrete foot points obtained by sampling; the smoothing method of the target trajectory is: two additional target points are set in the concave section of the trajectory, and the overall trajectory is made smoother by changing the envelope direction of the curve segment.

[0051] The target trajectory is smoothed because during reinforcement learning, the quadruped agent needs to maintain a certain level of exploratory activity in order to obtain higher reward values. Therefore, the trajectory it generates has many sharp points with abrupt changes in slope, which can cause huge acceleration changes and affect the walking stability of the closed-chain leg mechanism.

[0052] Target trajectory T such as Figure 6 As shown, it ultimately consists of 10 target points. T1 and T6 are two additional target points, and the other 8 target points are feature points on the original trajectory. The parameters of the target points and their corresponding time series information are shown in Table 1.

[0053] Table 1 Target point parameters and corresponding crank angles

[0054]

[0055] S42. Solving the foot position of the closed-chain leg mechanism.

[0056] The foot pose of the closed-loop leg configuration after screening is mainly solved using the vector loop method. In one embodiment, an XOY rectangular coordinate system is established with point D as the origin. The mechanism can be divided into 6 vector loops. By solving layer by layer, the mathematical mapping model between the crank input and the end output can be obtained. The vector expression of the specific loop is shown in equation (8):

[0057] (8)

[0058] Furthermore, taking the circuit DABC as an example, the relationship between the angle input of crank AB and the pose of hinge point C is solved, and the vector expression is decomposed using Euler decomposition:

[0059] (9)

[0060] in, , , , Let r1, r2, r3, and r4 be the angles between links DA, AB, BC, and CD and the positive X-axis, respectively; and let r1, r2, r3, and r4 be the lengths of links DA, AB, BC, and CD, respectively; and let r4 be the frame rod angles. Drive crank rotation angle And since the lengths r1, r2, r3, and r4 of each rod are known quantities, it is possible to determine... , Solve the problem.

[0061] Furthermore, the pose of hinge point C can be expressed by a complex equation:

[0062] (10)

[0063] Furthermore, the other loops can be solved in the same way to finally obtain the crank input angle. The relationship between the output of point I at the foot end and the output can be expressed as shown in equation (11):

[0064] (11)

[0065] S43. Design of reward function and constraints.

[0066] The reward function is represented by the target point T. i (i=0, 1, 2, ..., 9) and foot endpoint I i The negative of the sum of the Euclidean distances between them is given by equation (12):

[0067] (12)

[0068] Furthermore, the length variations of each link are shown in Table 2. In the first loop, namely the crank-four-bar linkage ABCD, to achieve continuous full revolutions of the crank, the mechanism must obey Grachoff's law, that is, the sum of the longest and shortest links is no greater than the sum of the other two links. This constraint can be expressed as:

[0069] (13)

[0070] Furthermore, through proper parameter design, the four-bar linkage can effectively avoid dead-point phenomena. Simultaneously, to improve transmission efficiency, the load on the connecting rods must be fully considered to prevent excessive loads that could lead to instability and fatigue damage. Therefore, it is necessary to impose appropriate constraints on the minimum transmission angle of the mechanism.

[0071] (14)

[0072] S44. Deep integration of deep reinforcement learning and size optimization of closed-chain leg mechanisms.

[0073] The deep reinforcement learning training environment is still based on the TD3 algorithm. The action space is the change in the length of each link, the state space is the length of each link, and the reward function is as shown in Equation (12). The closer the total Euclidean distance between the target point T and the optimized foot endpoint I, the higher the round reward value. After training, the output state value is the optimal link length of this closed-chain leg mechanism. The link length parameters after size optimization using deep reinforcement learning are shown in Table 2.

[0074] Furthermore, to highlight the advantages of using deep reinforcement learning for scale-based optimization, genetic algorithms and particle swarm optimization (PSO) were also selected to optimize the dimensional parameters of the mechanism. The reward function for both methods is the positional error between the target trajectory T and the foot trajectory I of the closed-loop mechanism, and the constraints are also consistent. The parameters for the genetic algorithm (GA) are set as follows: population size 200, maximum number of iterations 3000 generations, crossover probability 0.8, and roulette wheel selection strategy. The parameters for the particle swarm optimization (PSO) are set as follows: population size (number of particles) 100, maximum number of iterations 1000 generations, initial inertia weight 0.9, final inertia weight 0.4, individual learning factor 2.0, swarm learning factor 2.0, and maximum speed 20% of the parameter range. After optimization using deep reinforcement learning, genetic algorithm, and particle swarm optimization, the optimal dimensions of the leg mechanism are obtained, and corresponding foot trajectories are generated, such as... Figure 6 As shown.

[0075] This reveals that using deep reinforcement learning for mechanism size optimization results in more accurate synthesized optimal link length parameters and outputs foot trajectories that more closely match the target trajectory. In contrast, genetic algorithms and particle swarm optimization exhibit poor convergence and tend to overlook local details of the foot trajectory, leading to excessive errors. This analysis demonstrates the advantages of this invention in integrating deep reinforcement learning with the size optimization of closed-chain leg mechanisms.

[0076] exist Figure 7 The image shows the position-time curves and acceleration-time curves of the two trajectories in the x and y directions. It can be seen that the optimized trajectory effectively encompasses both the original and target trajectories, and exhibits good position-time fitting results in both the x and y directions. Furthermore, the peak error of the acceleration is only 1.2 m / s². 2This proves the effectiveness of the temporal dimension information of the adaptive bionic foot trajectory described in steps (1) and (2). It also shows that the lever length parameters optimized based on the temporal dimension information can ensure that the eight-bar linkage can effectively fit the velocity characteristics of the target trajectory. This further verifies the feasibility and innovation of using deep reinforcement learning for foot trajectory planning in this invention.

[0077] Table 2 Initial values ​​and optimized final rod length parameters

[0078]

[0079] Finally, to further verify the motion performance and practicality of the leg mechanism designed by the method proposed in this invention, a closed-chain octagonal mobile robot was built based on the optimized final rod length parameters in Table 2. Figure 9 As shown, the eight-legged design is solely for improved walking stability. It consists of four-legged units on both sides, and each unit's gait is similar to that of common quadruped robots, exhibiting a diagonal walking gait.

[0080] Its maximum center of mass fluctuation on flat ground is 13.42 mm, and its average forward speed is about 361.4 mm / s, demonstrating good stability and forward speed.

[0081] The above description is merely a preferred embodiment of this application and is not intended to limit the invention. Various modifications and variations can be made to this invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A closed-loop mobile robot leg design method based on deep reinforcement learning, characterized in that, include: S1. Build an open-chain eight-DOF bionic quadruped robot intelligent agent in a deep reinforcement learning training environment. After multiple iterations of training until convergence, it learns a motion control strategy that simulates the natural walking of animals. S2. Analyze the motion of the agent based on the principles of forward and inverse kinematics, and calculate the optimal foot trajectory during the agent's walking process; S3. Extract foot trajectory features from the foot trajectory, and select leg configurations in the closed kinematic chain atlas of mechanics based on the foot trajectory features; S4. After smoothing the above foot trajectory, input it into the closed-chain leg mechanism size optimizer to obtain the optimal size parameters of the leg configuration. The closed-chain leg mechanism size optimizer is developed using the method of optimizing the reward mechanism in deep reinforcement learning.

2. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 1, characterized in that, In step S1, a physical model and simulation training environment for the agent are established based on deep reinforcement learning technology. The agent is configured with a state space, action space, reward function, and training algorithm. After multiple iterations of training, the agent learns a motion control strategy that mimics natural walking.

3. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 2, characterized in that, The simulation training environment for the intelligent agent is built on the Matlab platform, and the physical model is created using Simulink modeling components. The state space of an intelligent agent includes the position, velocity, attitude angle, joint angle and torque of the body, and the action space has eight dimensions of information, which correspond to the rotational torque output by the eight joints of the four legs of the intelligent agent. The training algorithm for the intelligent agent is the TD3 algorithm; The agent's reward function consists of 5 sub-reward functions, and the expression for the total reward R is as follows: (1) in, This is the robot's forward reward value, used to ensure that the robot produces forward motion; This is the center of mass stability bonus value, used to limit the robot's trunk stability; This is the attitude angle reward value, used to measure the changes in the robot's roll angle, pitch angle, and yaw angle; To explore reward values, used to maintain the robot's exploration strategy; This is an energy consumption reward value, designed to minimize energy consumption during robot movement.

4. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 1, characterized in that, In step S2, the quadrupedal joint data of the trained agent are sampled at set time intervals. Then, the foot trajectory is solved using open-chain two-degree-of-freedom single-leg kinematics. The open-chain two-degree-of-freedom single-leg foot pose equation is as follows: (2) in This is the distance between the hip and knee joints. This is the distance between the foot and the knee joint. This refers to the hip joint rotation angle; This refers to the angle of rotation of the knee joint; The shape and temporal information of the foot trajectory are analyzed to extract the foot trajectory's span height, span distance, time period, and discrete points at the foot.

5. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 1, characterized in that, In step S3, after selecting the leg configuration from the closed kinetic chain diagram, the design is simplified to obtain a suitable closed-chain leg configuration.

6. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 5, characterized in that, In step S4, the input value of the closed-chain leg mechanism scale optimizer based on deep reinforcement learning is the target trajectory, and the output value is the optimal link length parameter of the leg mechanism. Its construction process includes: S41. Select the target trajectory based on the sampled discrete points at the foot end, and smooth the target trajectory. S42. Solve for the foot position of the closed-chain leg mechanism; S43. Set the reward function and constraints for the size optimization of the closed-loop leg mechanism; S44. Build a size optimization training environment based on deep reinforcement learning, configure the state space, action space, reward function and training algorithm. After training, the output state value is the optimal rod length of this closed-chain leg mechanism.

7. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 6, characterized in that, In step S41, the smoothing method of the target trajectory is as follows: additional target points are set in the concave segment of the trajectory, and the overall trajectory is made smoother by changing the envelope direction of the curve segment, so as to remove the sharp points of slope change in the foot trajectory generated by the agent.

8. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 6, characterized in that, In step S42, the foot pose of the screened closed-loop leg configuration is solved mainly by the vector loop method.

9. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 6, characterized in that, In step S43, the reward function is represented by the target point T. i (i=0, 1, 2, ..., 9) and foot endpoint I i The sum of the Euclidean distances between them; The constraints include: the length range of each link of the closed-chain leg mechanism, the existence condition of the crank, and the existence condition of the minimum transmission angle.

10. The method for designing the legs of a closed-loop mobile robot based on deep reinforcement learning according to claim 6, characterized in that, In step S44, a training environment is built based on the TD3 algorithm. The action space is the change in the length of each link, the state space is the length of each link, and the reward function is used to measure the quality of trajectory approximation. The closer the total Euclidean distance between the target point T and the optimized foot endpoint I, the higher the round reward value. After training is completed, the output state value is the optimal link length for this closed-chain leg mechanism.

Images