Spherical multi-telescopic-foot robot autonomous deformation motion method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of mobile robots, and particularly relates to a spherical multi-telescopic-foot robot autonomous deformation motion method based on deep reinforcement learning. The method comprises the following steps: constructing a state space and an action space based on motion characteristics of a robot; building a training scene in a simulation environment, and designing a narrow tunnel terrain and an unstructured terrain of a random height obstacle; constructing a deep reinforcement learning network model which comprises a strategy network, an action value function network, an experience pool and other core modules, and determining network parameters; designing a multi-target reward function according to task requirements; and training and optimizing the deep reinforcement learning network, and finally generating an efficient autonomous deformation motion strategy. Compared with the prior art, the robot has higher environmental adaptability and task generalization ability, and can autonomously adjust the deformation mode according to different terrain conditions, so that efficient and stable movement under the complex terrain is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile robots and relates to an autonomous deformation motion method of a spherical multi-telescopic foot robot based on deep reinforcement learning. Background Art

[0002] Terrain-adaptive robots are a branch of mobile robots that focus on movement on complex, uneven terrains and have strong environmental adaptability. In recent years, in order to achieve better terrain adaptability and motion stability, multi-legged robots have emerged, such as spherical multi-telescopic robots.

[0003] The spherical multi-telescopic leg robot exhibits high mobility through the design of spherical structure and retractable feet. Its multi-telescopic legs can independently adjust the length to adapt to different terrain features, thereby ensuring excellent mobility in complex environments such as rugged terrain, areas with dense obstacles or slopes. This design gives it the potential for wide application in scientific research and engineering fields, especially for tasks such as rescue and detection.

[0004] Deep reinforcement learning is a machine learning method that aims to optimize decision-making strategies through interaction with the environment so that the agent can maximize the cumulative reward in a specific task. In deep reinforcement learning, the agent gains experience through repeated trial and error, and continuously adjusts the strategy based on environmental feedback (i.e. rewards or penalties) to improve the effectiveness of task completion. Deep reinforcement learning is widely used in fields such as robot control and autonomous driving, and is particularly suitable for decision-making problems in complex dynamic environments.

[0005] CN117289693A discloses a path tracking method for a spherical multi-telescopic foot robot. The method constructs a dynamic model by establishing a kinematic model and calculating the contact force between the robot and the ground, and designs a control strategy for adjusting the movement direction according to the real-time position of the robot, so that the robot can flexibly select a suitable movement mode within or outside the predetermined path, and finally achieves tracking of a complex path. However, the method relies on a fixed control model, fails to dynamically adapt to the diversity of complex terrain, and lacks the ability of adaptive adjustment. Summary of the invention

[0006] In view of the shortcomings in the prior art, the present invention provides a spherical multi-telescopic leg robot autonomous deformation motion method based on deep reinforcement learning, thereby solving the problems that the traditional path control model is difficult to adapt to complex terrain, the real-time control is unstable, and the lack of adaptive adjustment capabilities.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A method for autonomous deformation motion of a spherical multi-telescopic leg robot based on deep reinforcement learning. The method for autonomous deformation motion of a spherical multi-telescopic leg robot. First, based on the robot's motion characteristics, the state space and action space are constructed; second, a training scene is built in a simulation environment, and an unstructured terrain including narrow tunnel terrain and random height obstacles is designed; third, a deep reinforcement learning network model is constructed, in which the modules include a policy network, an action value function network, a simulation environment and an experience pool, and its network parameters are determined; a multi-objective reward function is designed according to task requirements; finally, the deep reinforcement learning network is trained and optimized, and an efficient autonomous deformation motion strategy is finally generated. Specifically, the following steps are included:

[0009] S1: Based on the motion characteristics of the spherical multi-telescopic leg robot, the state space and action space are constructed. The state space includes the position and speed of the robot's telescopic joints, the speed and attitude angle of the central base, and the ground contact leg signal; the action space is the telescopic amount of each joint, forming a mapping relationship from state to action.

[0010] S2: Build a training environment in a simulation environment. Create an unstructured terrain including narrow tunnel terrain and random height obstacles. The spherical multi-telescopic leg robot relies on the body sensor to perceive the environment and collect perception information, including joint movements, joint extension and contraction, ground contact feedback signals, robot speed, and robot Euler angle.

[0011] S3: Design the network structure of the deep reinforcement learning algorithm. Determine the network parameters, including the policy network, action value function, network simulation environment and experience pool, and use soft update to update the local network parameters to the target network.

[0012] S4: Design a reward function according to different task environments. The reward function includes the robot's movement amount along the target direction, speed change amplitude, number of feet touching the ground, robot energy consumption, robot contact force penalty and collision penalty. The multi-objective reward function is used to guide the spherical multi-telescopic foot robot to achieve efficient and stable tumbling motion.

[0013] S5: Training deep reinforcement learning network model. The spherical multi-telescopic leg robot performs autonomous movement in a simulation environment, collects proprioceptive information as input state, generates corresponding quadruple, and stores it in the experience pool. The proprioceptive information includes joint movements, joint extension and contraction, and ground contact feedback signals. After accumulating enough minimum data in the experience pool to meet the training requirements according to the task settings, the parameters of the policy network and the action value function network are optimized by random sampling to gradually improve the policy performance. Before the maximum number of training rounds is reached, the target parameters of the policy network are periodically updated; when the training round reaches the set maximum number of rounds and the termination condition is met, the training is stopped and the final motion strategy is generated.

[0014] Furthermore, in step S1, through the design and analysis of the robot state space, the robot state space composition is obtained as follows: the unit length L of the 12 legs of the spherical multi-telescopic leg robot i 、The current extension speed V of the 12 legs i , robot ball center speed V C , the position of the robot's sphere center in the global coordinate system and the Euler angle S C , 12-foot touchdown signal information T i . Combining the above information into a column vector, we finally get a 45-dimensional state space:

[0015] S={L i ,V i ,V C ,S C ,T i}#(1)

[0016] For the robot's motion space, the robot uses position control in the simulation environment, and the controller directly outputs the target position of the robot's joints. Therefore, the robot's motion space is designed as the position l1…l of the robot's 12 legs. 12 , the position information is represented by data from 0 to 1:

[0017] {l1,…,l 12}#(2)

[0018] Further, in step S2, the simulation environment designs and builds a training environment, as follows:

[0019] S21: Create an S-shaped narrow tunnel terrain with a tunnel diameter of d S for:

[0020] d S =0.6d max #(3)

[0021] Among them, d max It is a spherical multi-telescopic-leg robot with a maximum diameter of 1 meter.

[0022] S22: Create an unstructured terrain and generate random height obstacles ranging from 0 to 5 cm for each point evenly distributed on the plane where the robot moves in the simulation environment.

[0023] Further, in step S3, the network structure of the deep reinforcement learning algorithm is designed as follows:

[0024] The TD3 (Twin Delayed Deep Deterministic policy gradient algorithm) algorithm is selected as the deep reinforcement learning algorithm. Its network structure includes a policy network, an action value function network No. 1, an action value function network No. 2, and their corresponding target networks, a total of 6 neural networks. The 6 neural networks have the same structure, all containing 2 hidden layers, and each hidden layer contains 256 neurons.

[0025] Further, in step S4, the reward function is designed as follows:

[0026]

[0027] Among them, R1 represents the displacement of the robot in the target direction at the current step as a positive reward; R2 represents the penalty for the robot's foot contacting the ground when it is greater than 3, limiting the robot's collision with the tunnel wall; R3 represents limiting the robot's total contact force; R4 represents the increase in the contact force at the current moment compared to the previous moment as a penalty; x now Represents the current displacement vector; x dir represents the target direction vector; ‖x dir ‖ represents the binary norm of the target direction vector; l represents the number of feet touching the ground; F now Represents the sum of the foot-leg contact forces at the current moment; F old represents the sum of the contact forces of the foot and leg at the last moment; ω1~ω4 represent the reward coefficients of each reward function. In order to train the robot to move across the curved tunnel to the end point, the current displacement vector x now and the target direction vector x dir The displacement of the robot's current step in the target direction is calculated as the positive reward R1; R3 and R4 limit the robot's violent collision with the wall or the ground to prevent the robot from making too intense movements.

[0028] The above four items R1 to R4 are basic rewards for measuring the robot's motion state. The total reward R is the sum of all rewards:

[0029]

[0030] Further, in step S5, the deep reinforcement learning network model is trained as follows:

[0031] S51: Start a round of training. The robot collects state information in the simulation environment as input information S t , where the state information includes joint movements, joint extension and contraction, ground contact feedback signals, robot speed, and robot Euler angles. The robot selects actions based on the following strategies:

[0032]

[0033] Among them, a t represents the action vector taken at time t; μ represents the policy network; S t represents the state information at time t; θ μ represents the policy network parameters; N t represents random noise; a random represents a random action; e represents the current round; e r represents the target round; P represents a random number, ranging from 0 to 1;

[0034] The current round e is less than the target round e r When , take random action a random Explore; the current round e is greater than the target round e r When the random number P from 0 to 1 is greater than 0.1, the action generated by the strategy μ plus the random noise N t As the action output, weak P is less than 0.1, then the random action a is still taken random Keep exploring.

[0035] S52: The robot moves in the simulation environment according to the action obtained in S51 to interact with the environment, and gives a reward according to the reward function. The environment gives the next state information S after the interaction. t+1 . Add the following four-tuple information to the experience pool. If the experience pool reaches the storage limit, delete the old data according to the first-in-last-out rule. If the amount of data in the experience pool is greater than the minimum size of each sampling, randomly sample a batch of data from the experience pool to update the policy network and action value function network. The four-tuple information is:

[0036] {S t , a t , reward, S t+1}#(7)

[0037] S53: Determine whether the maximum number of training steps is reached or whether the termination condition of this round of training is reached. If reached, start the next round of training until the predetermined training target round is reached to obtain the final exercise strategy.

[0038] Beneficial effects of the present invention:

[0039] (1) The present invention proposes a method for training motion strategies based on deep reinforcement learning technology, and constructs an end-to-end motion decision network that can directly output joint movements by relying solely on information provided by the robot's body sensors as input, without relying on complex intermediate steps such as kinematic modeling.

[0040] (2) The present invention obtains a highly robust motion strategy by appropriately and randomly changing the complex terrain environment in the simulation environment, and can adopt the most appropriate autonomous deformation motion mode to adapt to the environment according to different terrain conditions. This solves the problem that the traditional path control model relies on a precise dynamic model and is difficult to dynamically adapt to complex terrain. It has higher adaptability and generalization ability, and realizes efficient and stable autonomous motion control of the spherical multi-telescopic foot robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the autonomous deformation motion method of a spherical multi-telescopic leg robot based on deep reinforcement learning of the present invention;

[0042] Figure 2 This is one of the training scene diagrams in the simulation environment of the present invention;

[0043] Figure 3 The second is a training scene diagram in the simulation environment of the present invention;

[0044] Figure 4 It is a schematic diagram of the structure of the deep reinforcement learning algorithm of the present invention;

[0045] Figure 5 Schematic diagram of the training process of the deep reinforcement learning model;

[0046] Figure 6 It is the gait graph of movement in unstructured terrain; Figure 6 (a) is a sample of gait in unstructured terrain; Figure 6 (b) is a sample 2 of the gait in unstructured terrain; Figure 6 (c) is a sample of gait in unstructured terrain; Figure 6 (d) is a sample of gait in unstructured terrain;

[0047] Figure 7 This is the gait diagram for movement in narrow tunnel terrain; Figure 7 (a) is a sample 1 of the movement gait in a narrow tunnel terrain; Figure 7 (b) is a gait sample 2 in a narrow tunnel terrain; Figure 7 (c) is a sample 3 of the movement gait in a narrow tunnel terrain; Figure 7 (d) is a sample of the gait in a narrow tunnel. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1 As shown, the present invention comprises the following steps:

[0050] S1: Based on the robot's own motion characteristics, construct the state space and action space. The specific design process is as follows:

[0051] Through the design and analysis of the robot state space, the robot state space composition is obtained as follows: the unit length L of the 12 legs of the spherical multi-telescopic robot i ; Current extension speed V of the 12 legs i ; Robot ball center speed V C ; The position of the robot's sphere center in the global coordinate system and the Euler angle S C ; 12-foot touchdown signal information T i . Combining the above information into a column vector, we finally get a 45-dimensional state space:

[0052] S={L i ,V i ,V C ,S C ,T i}#(1)

[0053] For the robot's motion space, the robot uses position control in the simulation environment, and the controller directly outputs the target position of the robot's joints. Therefore, the robot's motion space is designed as the position l1…l of the robot's 12 legs. 12 , the position information is represented by data from 0 to 1:

[0054] {l1,…,l 12}#(2)

[0055] S2: Build a training environment in a simulation environment. The specific design process is as follows:

[0056] S21: Create a narrow S-shaped tunnel terrain:

[0057] like Figure 2 As shown, a narrow S-shaped tunnel terrain is created with a tunnel diameter d S for:

[0058] d S =0.6d max #(3)

[0059] Among them, d max The maximum diameter of the spherical multi-telescopic leg robot when its legs are fully extended is 1 meter. This terrain can verify the adaptability of the algorithm when the robot's range of activity is severely limited.

[0060] S22: Creating unstructured terrain:

[0061] like Figure 3 As shown, for 200 points evenly distributed on a plane with a width of 5 meters and a length of 4 meters, a random height obstacle ranging from 0 to 5 cm is generated for each point.

[0062] S3: Design the network model structure of the deep reinforcement learning algorithm:

[0063] The algorithm structure is as follows Figure 4 As shown in the figure, Actor represents the policy network; Critic1 represents the action value function network No. 1; Critic2 represents the action value function network No. 2; Actor_T, Critic_T1 and Critic_T2 represent the target networks of the above networks respectively; ENV represents the environment; Buffer represents the experience pool. The Actor network generates an action A with noise Noise according to the current state S, and passes it to the environment and the Critic network. The Critic1 and Critic2 networks evaluate the actions generated by the Actor respectively, calculate the corresponding losses Loss1 and Loss2, and thus guide the network optimization. The target network Actor_T and Critic_T1, Critic_T2 are responsible for adding noise Noise′ to the next time step state S′ to generate the next time step action A′ and the target value Q′, and combine the reward R returned by the environment to minimize the Bellman error and update the parameters of the Critic network. The entire architecture alleviates sample correlation through the experience pool to ensure stable optimization and efficient training of the strategy.

[0064] S4: Design reward functions according to different task environments:

[0065] The reward function includes the robot's movement along the target direction, the speed change, the number of feet touching the ground, the robot's energy consumption, the robot's contact force penalty and collision penalty. The multi-objective reward function is used to guide the robot to achieve efficient and stable tumbling motion.

[0066] S5: Train the deep reinforcement learning network model. The specific training process is designed as follows:

[0067] like Figure 5 As shown in Figure 1, the training process of deep reinforcement learning includes the following steps:

[0068] The first step is to initialize the training process and initialize the parameters of 6 neural networks (including the policy network, action value function network No. 1, action value function network No. 2 and their corresponding target networks).

[0069] The second step is to start a round of training, initialize the simulated virtual environment and set the initial position of the robot.

[0070] In the third step, the robot collects the current state information and selects random actions according to the strategy and executes the actions or generates actions through the strategy network, executes actions and interacts with the environment, and records the next time step state of the environment feedback and the corresponding reward value.

[0071] The fourth step is to determine whether the amount of data in the experience pool meets the minimum sample quantity requirement for policy update. If so, a batch of data is randomly sampled from the experience pool for training and updating the policy; then, the four-tuple data consisting of the current state, action, reward, and next state is stored in the experience pool. If the amount of data in the experience pool is insufficient, the above four-tuple data is directly stored in the experience pool without performing policy update.

[0072] Step 5: Check whether the maximum number of steps for this round of training has been reached or the preset termination condition has been met. If not, return to step 3 and continue to perform subsequent operations.

[0073] The sixth step is to determine whether the training has reached the set total training rounds. If not, return to the second step and start a new round of training; if reached, end the training and output the final strategy model.

[0074] The final motion strategy has a good effect on unstructured terrain and narrow tunnel terrain. Figure 6 Figure 7 As shown:

[0075] like Figure 6 As shown, Figure 6 (a) to Figure 6 (d) shows the robot's gait at different times in the unstructured terrain scene. The motion strategy generated by the present invention enables the robot to autonomously perform deformation motion by adjusting the length of its telescopic foot, thereby dynamically adapting to the randomly generated unstructured terrain and achieving a smooth and stable motion trajectory, reflecting that this motion strategy has strong adaptability and generalization ability in complex and unpredictable terrains.

[0076] like Figure 7 As shown, Figure 7 (a) to Figure 7 (d) shows the robot's gait at different times in a narrow tunnel. This scenario requires the robot to precisely control movement in a limited space. The example in the figure shows how the robot can flexibly cross the narrow part of the tunnel by adjusting the extension and posture of the foot, avoiding collisions and maintaining steady progress. The motion strategy generated by deep reinforcement learning enables the robot to autonomously select the optimal motion path under environmental constraints, reflecting the application advantage of this method in highly constrained terrain.

[0077] The above-described embodiments merely express the implementation methods of the present invention, but they should not be understood as limiting the scope of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for autonomous deformation motion of a spherical multi-telescopic foot robot based on deep reinforcement learning, characterized in that: The method comprises the following steps: Step S1: constructing a state space and an action space based on the motion characteristics of the spherical multi-telescopic leg robot; Step S2: Building a training environment in a simulation environment; creating an unstructured terrain including a narrow tunnel terrain and generating random height obstacles, and the spherical multi-telescopic leg robot relies on proprioceptive sensors to perceive the environment and collect proprioceptive perception information; Step S3: Design the network structure of the deep reinforcement learning algorithm; determine the network parameters, including the policy network, action value function, network simulation environment and experience pool, and use soft update to update the local network parameters to the target network; Step S4: designing a multi-objective reward function according to different task environments, and guiding the spherical multi-telescopic leg robot to achieve efficient and stable tumbling motion through the multi-objective reward function; Step S5: training a deep reinforcement learning network model; the spherical multi-telescopic-legged robot performs autonomous movement in a simulation environment, collects proprioceptive information as input state, generates corresponding quadruple, and stores it in an experience pool. After accumulating enough minimum data in the experience pool to meet the training requirements according to the task settings, the parameters of the policy network and the action value function network are optimized by random sampling to gradually improve the policy performance; before reaching the maximum number of training rounds, the target parameters of the policy network are periodically updated; when the training round reaches the set maximum number of rounds and the termination condition is met, the training is stopped and the final motion strategy is generated.

2. The method for autonomous deformation motion of a spherical multi-telescopic foot robot based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the state space includes the position and speed of the robot's telescopic joints, the speed and attitude angle of the center base, and the ground contact leg signal; the action space is the telescopic amount of each joint, forming a mapping relationship from state to action; In step S2, the proprioceptive information includes joint movements, joint extension and contraction amounts, ground contact feedback signals, robot speeds, and robot Euler angles; In step S4, the reward function includes the robot's movement amount along the target direction, the speed change amplitude, the number of feet touching the ground, the robot's energy consumption, the robot's contact force penalty and the collision penalty.

3. The method for autonomous deformation motion of a spherical multi-telescopic foot robot based on deep reinforcement learning according to claim 1, characterized in that: In step S1, through the design and analysis of the robot state space, the robot state space composition is obtained as follows: the unit length L of the 12 legs of the spherical multi-telescopic leg robot i 、The current extension speed V of the 12 legs i 、Robot ball center speed V C , the position of the robot's sphere center in the global coordinate system and the Euler angle S C , 12-foot touchdown signal information T i ; Combine the above information into a column vector, and finally get a 45-dimensional state space: S={L i ,V i ,V C ,S C ,T i }#(1) For the robot's motion space, the robot uses position control in the simulation environment, and the controller directly outputs the target position of the robot's joints; therefore, the robot's motion space is designed as the position l1…l of the robot's 12 legs. 12 , the position information is represented by data from 0 to 1: {l1,…,l 12}#(2)。 4. The method for autonomous deformation motion of a spherical multi-telescopic foot robot based on deep reinforcement learning according to claim 1, characterized in that: In step S2, the simulation environment designs and builds a training environment, as follows: Step S21: Create an S-shaped narrow tunnel terrain with a tunnel diameter d S for: d S =0.6d max #(3) Among them, d max The maximum diameter of the spherical multi-telescopic-leg robot is 1 meter; Step S22: Create an unstructured terrain, and generate a random height obstacle of 0 to 5 cm for each point evenly distributed on the plane where the robot moves in the simulation environment.

5. The method for autonomous deformation motion of a spherical multi-telescopic foot robot based on deep reinforcement learning according to claim 1, characterized in that: In step S3, the network structure of the deep reinforcement learning algorithm is designed as follows: The TD3 algorithm was selected as the deep reinforcement learning algorithm. Its network structure includes a policy network, an action value function network No. 1, an action value function network No. 2, and their corresponding target networks, a total of 6 neural networks; the 6 neural networks have the same structure, all containing 2 hidden layers, and each hidden layer contains 256 neurons.

6. The method for autonomous deformation motion of a spherical multi-telescopic foot robot based on deep reinforcement learning according to claim 1, characterized in that: In step S4, the multi-objective reward function is designed as follows: Among them, R1 represents the displacement of the robot in the target direction at the current step as a positive reward; R2 represents the penalty for the robot's foot touching the ground greater than 3 times, limiting the robot's collision with the tunnel wall; R3 represents limiting the robot's total contact force; R4 represents the increase in the contact force at the current moment compared to the previous moment as a penalty; x now Represents the current displacement vector; x dir represents the target direction vector; ‖x dir ‖ represents the binary norm of the target direction vector; l represents the number of feet touching the ground; F now Represents the sum of the foot-leg contact forces at the current moment; F old represents the sum of the contact forces of the foot and leg at the previous moment; ω1~ω4 represent the reward coefficients of each reward function; in order to train the robot's ability to cross the curved tunnel to the end point, the current displacement vector x now and the target direction vector x dir The displacement of the robot in the target direction at the current step is calculated as the positive reward R1; R3 and R4 limit the robot from violent collisions with the wall or the ground, preventing the robot from performing excessively violent movements; The above four items R1 to R4 are basic rewards for measuring the robot's motion state. The total reward R is the sum of all rewards:

7. The method for autonomous deformation motion of a spherical multi-telescopic foot robot based on deep reinforcement learning according to claim 1, characterized in that: In step S5, the deep reinforcement learning network model is trained as follows: Step S51: Start a round of training, the robot collects state information in the simulation environment as input information S t , where the state information includes joint movements, joint extension and contraction, ground contact feedback signals, robot speed, and robot Euler angles; the robot selects actions based on the following strategies: Among them, a t represents the action vector taken at time t; μ represents the policy network; S t represents the state information at time t; θ μ represents the policy network parameters; N t represents random noise; a random represents a random action; e represents the current round; e r represents the target round; P represents a random number, ranging from 0 to 1; When the current round e is less than the target round e r When , take random action a random Explore; when the current round e is greater than the target round e r When the random number P from 0 to 1 is greater than 0.1, the action generated by the strategy μ plus the random noise N t As the action output, weak P is less than 0.1, then the random action a is still taken random Keep exploring; Step S52: The robot moves in the simulation environment according to the action obtained in step S51 to interact with the environment, and gives a reward according to the reward function. The environment gives the next state information S after the interaction. t+1 ; Add the following four-tuple information to the experience pool. If the experience pool reaches the storage limit, delete the old data according to the first-in-last-out rule; if the amount of data in the experience pool is greater than the minimum size of each sampling, randomly sample a batch of data from the experience pool to update the policy network and action value function network; the four-tuple information is: {S t ,a t ,reward,S t+1 }#(7) Step S53: Determine whether the maximum number of training steps has been reached or whether the termination condition of this round of training has been reached. If so, start the next round of training until the predetermined training target round is reached to obtain the final exercise strategy.

Images