A snake-like robot control method based on reinforcement learning

Abstract

This invention discloses a snake-like robot control method based on reinforcement learning. The method includes: constructing a complex terrain simulation scene containing walls, cubic obstacles, and low passages; modeling the snake-like robot's structure, joints, and dynamic parameters to obtain a snake-like robot simulation model; defining the snake-like robot's state space and action space, and designing a diversified reward function; constructing an action network and a value network using a multi-layer perceptron (MLP) structure; and applying a reinforcement learning algorithm to conduct multiple rounds of simulation training on the snake-like robot, enabling the robot to learn parameter sets for making optimized decisions in different scenarios, thereby achieving optimal control of the snake-like robot and reaching the target point. This method does not rely on visual information and can achieve efficient and stable motion control of the snake-like robot in complex environments through autonomous learning, improving its ability to traverse low and narrow spaces.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a snake-like robot control method based on reinforcement learning. Background Art

[0002] Snake-like robots, due to their flexible structure and excellent environmental adaptability, have broad application prospects in complex terrain exploration, disaster relief, medical surgery, and other fields. Traditional snake-like robot control methods rely on pre-set motion patterns and complex motion planning algorithms. These methods often exhibit low adaptability and efficiency when faced with dynamic changes and unknown environments.

[0003] In recent years, reinforcement learning, a machine learning method that can autonomously learn optimal strategies through interaction with the environment, has been gradually applied to robotic control. However, effectively applying reinforcement learning to autonomous control of snake-like robots, especially enabling autonomous navigation in low, narrow spaces without relying on visual information, still faces challenges such as low algorithm efficiency, long training time, and insufficient model generalization.

[0004] Existing snake-like robot control methods have limited capabilities for autonomous navigation in complex environments, particularly in the absence of visual support, making efficient and stable motion control difficult. Furthermore, existing technologies often rely on complex sensor systems and high computational costs when navigating low, narrow spaces such as holes, resulting in high system complexity and cost, and limited adaptability. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a snake-like robot control method based on reinforcement learning. This method does not rely on visual information and is controlled based on joint status and position sensor information. It can achieve efficient and stable motion control of the snake-like robot in complex environments through autonomous learning, and improve its ability to traverse low and narrow spaces.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A snake-like robot control method based on reinforcement learning comprises the following steps:

[0008] Step 1: Build a complex terrain simulation scene containing walls, cubic obstacles, and low passages;

[0009] Step 2: Model the structure, joints and dynamic parameters of the snake-like robot to obtain a simulation model of the snake-like robot;

[0010] Step 3: Define the state space and action space of the snake robot, and design a diversified reward function that comprehensively considers the speed of the snake robot's head moving towards the target direction, the degree of alignment between the movement direction and the target direction, the change in distance to the target point, whether a collision occurs, and the amplitude and change of joint movements;

[0011] Step 4: Use the multi-layer perceptron (MLP) structure to build the action network and value network;

[0012] Step 5: Based on steps 1 to 4, the snake robot is trained with a reinforcement learning algorithm for multiple rounds of simulations. This allows the robot to learn the parameter sets required to make optimal decisions in different scenarios, thereby achieving optimal control of the snake robot and helping it reach the target point.

[0013] To optimize the above technical solutions, specific measures taken also include:

[0014] In the simulation environment described in step 1 above, multiple light sources are set to ensure the visualization of the robot and obstacles, and to assist in debugging and observing the simulation effect.

[0015] The state space in step 3 includes the snake robot's joint angles, joint velocities, snake head position, the distance between the snake robot and the target point, and the position of the obstacle; the action space is the control signal of each joint of the snake robot.

[0016] The diversified reward function described in step 3 above is: R t =R forward +R angle +R distance +R target +R collision +R cost , where the forward reward R forward Rewards are given based on the speed at which the snake's head moves towards the target; angle reward R angle Rewards are given based on the degree of alignment between the snake's movement direction and the target direction; distance change reward R distance Reward or penalty is given based on the change in the distance from the snake head to the target point; the target distance penalty R target Penalty is given based on the distance from the snake head to the target point; reward R is given for achieving the target. collision By judging whether the snake head has successfully reached the target point, a penalty is given; the cost penalty R is controlled cost Penalties are imposed based on the magnitude and variability of joint movement.

[0017] The above R forward The calculation formula is: forward =w forward ·v goal , where v goal 、w forwardare the component and weight coefficient of the snake head velocity in the target direction respectively;

[0018] R angle The calculation formula is: angle = cos(θ), where θ is the angle between the snake's head and the target direction;

[0019] R distance The calculation formula is: Among them, Δd represents the change in the distance from the snake head to the target;

[0020] R target The calculation formula is: Among them, d represents the distance from the current snake head to the target point, d threshold is the distance threshold, w1 is the weight coefficient;

[0021] R collision The calculation formula is: Among them, C collision is a constant;

[0022] R cost Indicates the resources consumed by the robot to perform actions. The calculation formula is: R cost =w2·∑ i |action i | 2 , where w2 is the weight coefficient of control cost, action i is the size of the ith action.

[0023] The input layer of the action network described in the above step 4 receives state information from the environment, including joint angles, joint velocities, snake head positions, and obstacle positions; the hidden layer contains two layers, each containing 256 neurons, and the activation function uses ReLU; the output layer outputs the control signal of each joint, and uses the tanh activation function to map the output to the control range; and uses log_std_parameter to control the exploratory nature of the action, and samples the action through Gaussian distribution to enhance the robustness of the strategy; the value network is used to evaluate the value function of the current state and assist the strategy network in optimizing its action selection strategy. Its input layer receives state information from the environment, including joint angles, joint velocities, snake head positions, and obstacle positions, and the output layer outputs the value estimate of the current state.

[0024] In each training cycle of step 5 above, the snake robot interacts with the environment and collects state s t 、Action a t , reward R t The data is stored in the memory bank, and the reinforcement learning algorithm uses the data in the memory bank to calculate the clipping objective function and the value loss function.CLIP (θ) updates the parameters θ of the policy network so that the robot can choose better actions to maximize the cumulative reward. At the same time, the value network minimizes the value loss function L VF (φ), updates the value parameter φ to accurately estimate the value of each state and provide effective guidance for the policy network. In the snake robot control task, the total loss function L(θ,φ) is used to ensure the effective training of the policy network and the value network, and the efficiency and stability of its movement are ensured by stabilizing the policy update.

[0025] The above clipping objective function is as follows:

[0026]

[0027] The value loss function is as follows:

[0028]

[0029] The total loss function is as follows:

[0030] L(θ,φ)=L CLIP (θ)-c1L VF (φ)+c2S[π θ ](s t )

[0031] in,

[0032] represents the average value of all sampled time steps t;

[0033] is the advantage function;

[0034] Clip(r t (θ),1-∈,1+∈) represents the strategy ratio r t (θ) is limited to the range [1-∈, 1+∈];

[0035] ∈ is a constant;

[0036] π θ (a t |s t ) is the current policy network in a given state s t Next select action a t The probability density function of

[0037] For the old policy network in a given state s t Next select action a t The probability density function of

[0038] V φ (st ) is in state s t Estimation of the value function under ;

[0039] is the target value at time step t;

[0040] S[π θ ](s t ) is in state s t Next strategy π θ Entropy;

[0041] r t (θ) is the policy ratio of t rounds of training;

[0042] c1, c2 are hyperparameters.

[0043] Snake robots need to move smoothly in complex environments, avoiding drastic changes in motion caused by large policy updates. The clipping objective function ensures smooth and continuous motion by limiting the magnitude of policy updates.

[0044] The present invention has the following beneficial effects:

[0045] The present invention does not rely on visual information, but is mainly controlled based on non-visual information such as joint angles, joint speeds, snake head position and obstacle position. It does not rely on cameras or other visual sensors, which significantly reduces the complexity and cost of the system and improves its adaptability in visually restricted environments, enabling the robot to adapt to different environments and challenges, and improving its flexibility and success rate in specific task scenarios.

[0046] This invention uses a diversified reward function to guide the snake robot to optimize its motion strategy, achieving efficient forward speed and stable motion posture. This ensures smooth movement in complex environments and avoids collisions or loss of control caused by unstable motion. It also effectively identifies and avoids obstacles in the environment, optimizing the snake robot's path planning through low and narrow passages, thereby improving the robot's navigation capabilities and task completion rate in complex environments.

[0047] During the training process, the present invention improves the generalization ability of the model by introducing diverse simulation environment configurations, such as obstacles and low passages of different positions and sizes, so that the trained control strategy can adapt to a variety of complex environments, thereby enhancing the practicality and flexibility of the system.

[0048] The present invention provides an efficient and adaptable control method by combining a reinforcement learning algorithm with the autonomous control of a snake-like robot. Through autonomous learning, the snake-like robot can achieve efficient and stable motion control in complex environments, especially through low and narrow spaces such as holes, from one end to the other without the need for visual information. This significantly improves the snake-like robot's ability to move in complex environments, especially through low and narrow spaces without the need for visual information.

[0049] The core purpose of clipping the objective function in this paper is to control the amplitude of policy changes and prevent drastic changes during policy updates, thereby ensuring training stability and convergence. Unlike traditional clipping methods, this paper introduces a dynamic adjustment mechanism that dynamically adjusts the clipping ratio ∈ based on the performance and convergence of the current training phase, thereby better controlling the amplitude of policy updates.

[0050] The value loss function aims to optimize the value network's estimate of state value. In motion control tasks in low, narrow spaces, the value loss function enhances the value network's assessment of individual actions through reward correction. The total loss function adjusts the contribution of each component through weights to ensure coordination and balance during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flowchart of the snake-like robot control method based on reinforcement learning of the present invention;

[0052] Figure 2 This is a schematic diagram of the process of the present invention using a snake-like robot based on reinforcement learning to pass through a low and narrow passage;

[0053] Figure 3 This is a schematic diagram of the process of the reinforcement learning-based snake robot passing through a low and narrow passage and reaching the target location. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] Although the steps in the present invention are arranged with numbers, they are not intended to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0056] The present invention provides a snake robot control method based on reinforcement learning. The method includes constructing a snake robot simulation model, setting up a reinforcement learning training environment, obtaining the current state information of the snake robot through joint state and position sensors, and selecting actions through a value network to achieve the goal of passing through low and narrow passages. The reinforcement learning algorithm PPO is applied, and through iterative training, the robot learns the parameter set for making optimized decisions in different scenarios. This method enables the robot to adapt to different environments and challenges, and improves its flexibility and success rate in specific task scenarios. Figure 1 As shown, the method specifically includes the following steps:

[0057] Step 1: Build a complex terrain simulation scene containing walls, cubic obstacles, and low passages;

[0058] Step 2: Model the structure, joints and dynamic parameters of the snake-like robot to obtain a simulation model of the snake-like robot;

[0059] Step 3: Define the state space and action space of the snake robot, and design a diversified reward function that comprehensively considers the speed of the snake robot's head moving towards the target direction, the degree of alignment between the movement direction and the target direction, the change in distance to the target point, whether a collision occurs, and the amplitude and change of joint movements;

[0060] Step 4: Use the multi-layer perceptron (MLP) structure to build the action network and value network;

[0061] Step 5: Based on steps 1 to 4, the snake robot is trained with a reinforcement learning algorithm for multiple rounds of simulations. This allows the robot to learn the parameter sets required to make optimal decisions in different scenarios, thereby achieving optimal control of the snake robot and helping it reach the target point.

[0062] The embodiments cover multiple aspects, including the construction of a simulation environment, the design and implementation of a reinforcement learning environment, the training and optimization of a reinforcement learning model, and the testing and verification of the model, as follows:

[0063] like Figure 2 As shown in the figure, step one creates a simulation scene with complex terrain, including walls, cubic obstacles, and low passages. The simulation environment is modeled and simulated using the MuJoCo simulation platform. A plane-type geometry is used as the ground, and walls are set up on both sides of the environment to prevent the snake robot from exceeding the predetermined area and guide it through the low passage. Multiple cubic obstacles are placed in the environment to form a narrow passage, ensuring that the snake robot must pass through the low passage to continue moving forward. At the same time, the shortest path to the target point requires the narrowest passage, thus increasing the challenge of the snake robot navigating the low passage.

[0064] Using the XML configuration file above, we created a simulation model for the snake robot and incorporated walls, cubic obstacles, and low passages into the environment. By adjusting the size and position of each geometric object, we ensured that the simulation environment simulated the complex terrain encountered in real-world applications. In particular, the size and position of the low passages were optimized to ensure the snake robot could navigate smoothly.

[0065] Multiple light sources are set in the simulation environment to ensure good visualization of the robot and obstacles, and to assist in debugging and observing the simulation effect.

[0066] Step 2: Model the structure, joints and dynamic parameters of the snake-like robot in detail to obtain a simulation model of the snake-like robot;

[0067] The snake-like robot comprises a snake body connected by multiple joints, each joint being controlled by a corresponding actuator. The robot body comprises a snake head, a snake body connected by multiple joints, and a snake tail.

[0068] Step three designs the state space of the deep reinforcement learning algorithm, including information such as joint angles, joint velocities, snake head position, distance from the target point, and the position of obstacles. In the embodiment, the snake robot includes angles (12) and angular velocities (12) of all joints, a total of 24 values, reflecting the current posture and motion state of the snake robot. The position state mainly includes the position coordinates (x, y) of the snake head in the two-dimensional plane, and the distance from the target point, which are used to guide the snake robot to move in the target direction. The obstacle information includes four cubic obstacles, and the height is limited by the top cube.

[0069] The action space is defined as the control signal for each joint. In this embodiment, for each joint of the snake robot, 12 continuous values ​​are defined, each corresponding to a joint control signal within the range [-1, 1]. The control signal is converted to the actual control range of the actuator [-7, 7] through linear mapping.

[0070] Reward functions are key to guiding reinforcement learning agents to learn correct behaviors. This paper designs a diversified reward function that comprehensively considers factors such as the snake robot's speed toward the target, the degree of alignment between the movement direction and the target direction, the reward for navigating low passages, the penalty for obstacle avoidance, and the cost of controlling energy consumption, to guide the robot in learning the optimal movement strategy. The reward function includes a forward movement reward, an angle reward, a distance change reward, a target distance penalty, and a goal achievement reward. These functions encourage the snake robot to move toward the target, avoid collisions, and successfully navigate low passages, guiding it to learn a strategy for efficiently navigating low passages and avoiding obstacles.

[0071] The current calculation method of the diversification reward value Rt is as follows:

[0072] R t =R forward +R angle +R distance +R target +R collision +R cost

[0073] The forward reward is based on the speed of the snake head moving towards the target direction, which is mainly calculated as the component v of the snake head speed in the target direction. goal , multiplied by the weight coefficient w forward The specific formula is as follows:

[0074] R forward =w forward ·v goal

[0075] The angle reward is the degree of alignment between the snake's movement direction and the target direction. This is done by calculating the angle θ between the snake's actual movement direction and the target direction, taking the cosine value and multiplying it by the weight coefficient. The closer the movement direction is to the target direction, the greater the reward. When the snake's movement direction is consistent with the target direction, the reward is maximum (cos(0) = 1); when the direction is opposite, the reward is minimum (cos(π) = -1). This is shown in the following formula:

[0076] R angle =cos(θ)

[0077] Distance change rewards are mainly based on the change in the distance from the snake head to the target point. If the current distance is reduced compared to the previous moment, a reward is given; if it increases, a penalty is given. Δd represents the change in the distance to the target. The specific formula is as follows:

[0078]

[0079] Δd=d previous -d current

[0080] The target distance penalty is based on the distance from the snake head to the target point, which encourages the robot to approach the target. When the distance exceeds the set threshold of 0.1 meters, a negative reward proportional to the distance is given, and the maximum penalty value is set to -1.0. d represents the current distance from the snake head to the target point, d threshold is the distance threshold, which is set to 0.1 here, representing the radius of the green target ball in the simulation. The specific formula is as follows:

[0081]

[0082] Among them, w1 is the weight coefficient.

[0083] By checking the contact with the obstacle, it is determined whether a collision occurs. If a collision occurs, an additional negative reward self._collision_penalty is given. The passage range is defined according to the pos and size of low_passage in the XML. The collision is determined by checking whether the snake head position is within the predetermined range of the low passage. The specific formula is as follows:

[0084]

[0085] Among them, C collision is a constant.

[0086] Penalties are imposed based on the magnitude and variation of joint motions to encourage energy conservation and stable control. The method is to calculate the control cost penalty R cost Represents the resources consumed by the robot to perform actions. Large changes in joint angles require more energy, while small changes in angles consume less energy. Its calculation formula is: R cost =w2·∑ i |action i | 2 , where w2 is the weight coefficient of control cost, action i is the size of the ith action.

[0087] The above rewards and penalties are weighted and superimposed to form the final reward signal.

[0088] Step 4: Use the multi-layer perceptron (MLP) structure to build the action network and value network.

[0089] The network architecture uses a multi-layer perceptron (MLP) structure to establish an action neural network (Actor). This network is designed to generate appropriate control signals based on the current environmental state, guiding the snake robot to perform effective locomotion, particularly when navigating low and narrow spaces with high efficiency and stability. Its input layer receives state information from the environment, including joint angles, joint velocities, snake head position, and obstacle locations, with a total input dimension of 40. The hidden layer consists of two layers, each with 256 neurons. Reluctant Unit (ReLU) activation functions enhance the network's nonlinear representation and ensure it effectively captures complex relationships between state and action. The output layer outputs control signals for each joint. The tanh activation function maps the outputs to a control range, constraining the output actions to the range [-1, 1] to accommodate the control requirements of each joint. The log_std_parameter is then used to control the exploratory nature of the actions, and actions are sampled using a Gaussian distribution to enhance the robustness of the policy.

[0090] A value neural network, the Critic, is established. This network evaluates the value function of the current state and assists the policy network in optimizing its action selection strategy. The value network also uses a multilayer perceptron (MLP) structure to ensure accurate value estimation. Its input layer receives the same state information as the policy network, and its output layer outputs the value estimate of the current state.

[0091] Step 5: Training and implementing the reinforcement learning model: Using a reinforcement learning algorithm to train the snake robot, through multiple rounds of simulation training, the robot gradually optimizes its motion strategy, enabling it to autonomously avoid obstacles and pass through low passages in extremely narrow and complex environments.

[0092] Specifically, the current joint angles, joint velocities, snake head positions, and obstacle positions are obtained from the simulation environment to form a state vector. The reinforcement learning agent selects an action based on the current state. The action is converted into a control signal of the actuator through linear mapping and applied to the simulation environment.

[0093] According to the set rendering mode human, the movement state of the snake robot is visualized in real time, which is convenient for observation and debugging.

[0094] By running simulations, we ensure that the positions and dimensions of all obstacles and passages in the environment are set correctly, and the snake robot can move freely in the environment.

[0095] Use visualization tools to check the interaction between the snake robot and obstacles to ensure that collision detection is working correctly.

[0096] The performance of the training model is verified through simulation tests and actual applications. The reward function weight, algorithm parameters and model structure are adjusted according to the evaluation results to improve the stability and generalization ability of the control method.

[0097] In order to achieve autonomous control of the snake-like robot in complex environments, the present invention uses the Proximal Policy Optimization (PPO) algorithm in the Stable Baselines3 library to train the snake-like robot and optimize its autonomous control capability in complex environments.

[0098] The proposed PPO algorithm, based on the policy gradient method, ensures the stability of policy updates by limiting the magnitude of policy updates. The PPO algorithm updates the policy parameter θ by minimizing the following value loss function and updates the value parameter φ by minimizing the following value loss function. The following formula is combined with the control of the snake robot to achieve efficient and stable autonomous navigation and obstacle avoidance. In the snake robot control task, the total loss function L(θ, φ) ensures effective training of the policy and value networks, and stabilizes policy updates to ensure efficient and stable motion.

[0099] The specific formula for the strategy ratio is as follows, where the numerator is the action probability density function output by the current strategy network, and the denominator is the action probability density function of the old strategy, which is used to measure the difference between the new and old strategies in the current action selection.

[0100]

[0101] The specific formula of the clipping objective function is as follows:

[0102]

[0103] The specific formula of the value loss function is as follows:

[0104]

[0105] The specific formula of the total loss function is as follows:

[0106] L(θ,φ)=L CLIP (θ)-c1L VF (φ)+c2S[π θ ](s t )

[0107] in It means taking the average of all sampled time steps t and performing expected calculation on the data of the entire batch to ensure the stability and consistency of the strategy update.

[0108] is the advantage function, calculated by the generalized advantage estimation (GAE) method, representing action a t In state s t The advantage of the next state relative to the current state. The calculation formula is as follows:

[0109] A t =R t -V φ (s t )

[0110] Clip is to limit the strategy ratio rt(θ) to the range of [1-∈, 1+∈], so that the strategy is updated in a favorable direction and the advantage function is used. To guide optimization.

[0111] ∈ is a small constant that controls the extent of clipping.

[0112] π θ (a t |s t ) is the current policy network in a given state s t Next select action a t The probability density function of .

[0113] For the old policy network in a given state s t Next select action a t The probability density function of .

[0114] V φ (s t ) is in state s t The value function estimation under .

[0115] is the target value at time step t, used to train the value network.

[0116] S[π θ ](s t ) is in state s t Next strategy π θ The entropy of the strategy is used to measure the state s t The randomness of the following.

[0117] c1 and c2 are hyperparameters used to balance the value loss and entropy regularization terms, respectively.

[0118] In each training cycle, the snake robot interacts with the environment and collects the state s t 、Action a t , reward Rt and other data, and store them in the memory bank.

[0119] The PPO algorithm uses the aforementioned formula to calculate the loss function and update the policy network's parameters θ, enabling the robot to select optimal actions to maximize its cumulative reward. Simultaneously, the value network updates its parameters φ by minimizing the value loss function to accurately estimate the value of each state, providing effective guidance for the policy network.

[0120] The learning rate is set to 3e-4 to balance learning speed and stability, ensuring that the model can learn effectively without causing training instability due to excessively high learning rates. The discount factor is set to 0.99 to consider and emphasize long-term rewards, which helps the snake robot to plan for the long term in complex environments. The number of policy updates is set to 2048 to ensure sufficient samples are collected before each policy update, improving training efficiency.

[0121] Use the check_env function to ensure that the environment meets the OpenAI Gym interface requirements. Then, call the learn method with a total number of training steps set to 5,000,000 to start the reinforcement learning training process.

[0122] Use the PPO.load method to load the previously saved training model. Before the test begins, reset the simulation environment and initialize the snake robot's state. Use the trained model to predict actions and apply the predicted actions to the simulation environment. Use real-time visualization to observe whether the snake robot can successfully pass through the low channel and avoid all obstacles.

[0123] After the training is completed, the snake robot is tested using the trained model to verify its ability to pass through low passages in complex environments.

[0124] Depend on Figure 2 It can be clearly seen that in the simulation environment, the method proposed in the present invention successfully guides the snake-like robot through the low passage.

[0125] Depend on Figure 3 It can be clearly seen that in the simulation environment, the method proposed in the present invention successfully guides the snake-like robot through the low passage and then reaches the target position.

[0126] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0127] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A snake-like robot control method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Build a complex terrain simulation scene containing walls, cubic obstacles, and low passages; In the simulation scenario, walls are set up on both sides of the environment to prevent the snake robot from exceeding the predetermined area and to guide it through a low passage. Multiple cubic obstacles are placed in the environment to form a narrow passage, ensuring that the snake robot needs to pass through the low passage to continue moving forward. At the same time, the shortest path to the target point requires passing through the narrowest passage, thereby increasing the challenge for the snake robot to pass through the low passage. Step 2: Model the structure, joints and dynamic parameters of the snake-like robot to obtain a simulation model of the snake-like robot; Step 3: Define the state space and action space of the snake robot, and design a diversified reward function that comprehensively considers the speed of the snake robot's head moving towards the target direction, the degree of alignment between the movement direction and the target direction, the change in distance to the target point, whether a collision occurs, and the amplitude and change of joint movements; The diversified reward function is: R t =R forward +R angle +R distance +R target +R collision +R cost , where the forward reward R forward Rewards are given based on the speed at which the snake's head moves towards the target; angle reward R angle Rewards are given based on the degree of alignment between the snake's movement direction and the target direction; distance change reward R distance Reward or penalty is given based on the change in the distance from the snake head to the target point; the target distance penalty R target Penalty is given based on the distance from the snake head to the target point; reward R is given for achieving the target. collision By judging whether the snake head has successfully reached the target point, a penalty is given; the cost penalty R is controlled cost Penalties are imposed based on the magnitude and variability of joint movements; The R forward The calculation formula is: forward =w forward ·v goal , where v goal 、w forward are the component and weight coefficient of the snake head velocity in the target direction respectively; R angle The calculation formula is: angle = cos(θ), where θ is the angle between the snake's head and the target direction; R distance The calculation formula is: Among them, Δd represents the change in the distance from the snake head to the target; R target The calculation formula is: Among them, d represents the distance from the current snake head to the target point, d threshold is the distance threshold, w1 is the weight coefficient; R collision The calculation formula is: Among them, C collision is a constant; R cost Indicates the resources consumed by the robot to perform actions. The calculation formula is: R cost =w2·∑ i |action i | 2 , where w2 is the weight coefficient of control cost, action i is the size of the ith action; Step 4: Use the multi-layer perceptron (MLP) structure to build the action network and value network; Step 5: Based on steps 1 to 4, the snake robot is trained with a reinforcement learning algorithm for multiple rounds of simulations. This allows the robot to learn the parameter sets required to make optimal decisions in different scenarios, thereby achieving optimal control of the snake robot and helping it reach the target point.

2. A snake-like robot control method based on reinforcement learning according to claim 1, characterized in that: In the simulation environment described in step 1, multiple light sources are set up to ensure the visualization of the robot and obstacles, and to assist in debugging and observing the simulation effect.

3. The snake-like robot control method based on reinforcement learning according to claim 1, characterized in that: The state space in step 3 includes the joint angles, joint velocities, snake head position, distance between the snake robot and the target point, and the position of the obstacle; the action space is the control signal of each joint of the snake robot.

4. The snake-like robot control method based on reinforcement learning according to claim 1, characterized in that: In step 4, the input layer of the action network receives state information from the environment, including joint angles, joint velocities, snake head positions, and obstacle positions; the hidden layer contains two layers, each containing 256 neurons, and the activation function uses ReLU; the output layer outputs the control signal of each joint, and uses the tanh activation function to map the output to the control range; and uses log_std_parameter to control the exploratory nature of the action, and samples the action through Gaussian distribution to enhance the robustness of the strategy; the value network is used to evaluate the value function of the current state and assist the strategy network in optimizing its action selection strategy. Its input layer receives state information from the environment, including joint angles, joint velocities, snake head positions, and obstacle positions, and the output layer outputs the value estimate of the current state.

5. The snake-like robot control method based on reinforcement learning according to claim 1, characterized in that: In each training cycle of step 5, the snake robot interacts with the environment and collects state s t 、Action a t , reward R t The data is stored in the memory bank, and the reinforcement learning algorithm uses the data in the memory bank to calculate the clipping objective function and the value loss function. CLIP (θ) updates the parameters θ of the policy network so that the robot can choose better actions to maximize the cumulative reward. At the same time, the value network minimizes the value loss function L VF (φ), updates the value parameter φ to accurately estimate the value of each state and provide effective guidance for the policy network. In the snake robot control task, the total loss function L(θ,φ) is used to ensure the effective training of the policy network and the value network, and the efficiency and stability of its movement are ensured by stabilizing the policy update.

6. The snake-like robot control method based on reinforcement learning according to claim 5, characterized in that: The clipping objective function is as follows: The value loss function is as follows: The total loss function is as follows: L(θ,φ)=L CLIP (θ)-c1L VF (φ)+c2S[π θ ](s t ) in, Indicates taking the average value of all sampled time steps t; is the advantage function, representing action a t In state s t The advantage of the next state relative to the current state is calculated as follows: A t =R t -V φ (s t ) Clip(r t (θ),1-∈,1+∈) represents the strategy ratio r t (θ) is limited to the range [1-∈, 1+∈]; ∈ is a constant; π θ (a t |s t ) is the current policy network in a given state s t Next select action a t The probability density function of For the old policy network in a given state s t Next select action a t The probability density function of V φ (s t ) is in state s t Estimation of the value function under ; is the target value at time step t; S[π θ ](s t ) is in state s t Next strategy π θ Entropy; r t (θ) is the policy ratio of t rounds of training; c1, c2 are hyperparameters.

Images