Snakelike robot control method based on reinforcement learning

Abstract

The invention discloses a snake-shaped robot control method based on reinforcement learning. The snake-shaped robot control method comprises the steps that a complex terrain simulation scene comprising walls, cubic obstacles and low channels is built; modeling is conducted on the structure, joints and kinetic parameters of the snake-shaped robot, and a snake-shaped robot simulation model is obtained; defining a state space and an action space of the snakelike robot, and designing a diversified reward function; constructing an action network and a value network by adopting a multi-layer perceptron (MLP) structure; and performing multi-round simulation training on the snakelike robot by applying a reinforcement learning algorithm, so that the robot learns a parameter set for making an optimization decision under different scenes, thereby realizing optimization control of the snakelike robot and reaching a target point. According to the method, visual information does not need to be depended on, efficient and stable motion control of the snake-shaped robot in a complex environment can be achieved through autonomous learning, and the crossing capacity of the snake-shaped robot in a low and narrow space is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a snake-like robot control method based on reinforcement learning. Background Art

[0002] Snake robots have broad application prospects in complex terrain exploration, disaster relief, medical surgery, etc. due to their flexible structure and excellent environmental adaptability. Traditional snake robot control methods mostly rely on preset motion patterns and complex motion planning algorithms, which often show low adaptability and efficiency when facing dynamic changes and unknown environments.

[0003] In recent years, reinforcement learning, a machine learning method that can autonomously learn optimal strategies through interaction with the environment, has gradually been applied to robot control. However, effectively applying reinforcement learning to the autonomous control of snake-like robots, especially to achieve autonomous navigation of snake-like robots in low and narrow spaces without relying on visual information, still faces challenges such as low algorithm efficiency, long training time, and insufficient model generalization ability.

[0004] Existing snake robot control methods have limited autonomous navigation capabilities in complex environments, especially in the absence of visual information support, making it difficult to achieve efficient and stable motion control. In addition, existing technologies often rely on complex sensor systems and high computing costs when dealing with low and narrow spaces such as holes, resulting in high system complexity and cost, and insufficient adaptability. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a control method for a snake-like robot based on reinforcement learning in response to the deficiencies of the above-mentioned prior art. The method does not rely on visual information and is controlled based on joint status and position sensor information. It can realize efficient and stable motion control of the snake-like robot in complex environments through autonomous learning, and improve its ability to traverse low and narrow spaces.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A snake-like robot control method based on reinforcement learning comprises the following steps:

[0008] Step 1: Build a complex terrain simulation scene including walls, cube obstacles and low passages;

[0009] Step 2: Model the structure, joints and dynamic parameters of the snake-like robot to obtain a simulation model of the snake-like robot;

[0010] Step 3: Define the state space and action space of the snake robot, and design a diversified reward function by comprehensively considering the movement speed of the snake robot's head toward the target direction, the degree of alignment between the movement direction and the target direction, the change in the distance to the target point, the distance to the target point, whether a collision occurs, and the amplitude and change of joint movements;

[0011] Step 4: Use the multi-layer perceptron MLP structure to build the action network and value network;

[0012] Step 5: Based on steps 1 to 4, a reinforcement learning algorithm is used to conduct multiple rounds of simulation training on the snake robot, so that the robot can learn the parameter sets for making optimal decisions in different scenarios, realize the optimal control of the snake robot, and reach the target point.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] In the above step 1, multiple light sources are set in the simulation environment to ensure the visualization of the robot and obstacles, and to assist in debugging and observing the simulation effect.

[0015] The state space in the above step 3 includes the joint angles, joint velocities, snake head position, distance between the snake robot and the target point, and the position of obstacles of the snake robot; the action space is the control signal of each joint of the snake robot.

[0016] The diversified reward function described in step 3 above is: R t =R forward +R angle +R distance +R target +R collision +R cost , where the forward reward R forward Rewards are given based on the speed at which the snake head moves towards the target direction; angle rewards R angle Rewards are given based on the alignment between the movement direction of the reward snake head and the target direction; distance change reward R distance Reward or penalty is given based on the change in the distance from the snake head to the target point; target distance penalty R target Penalty is given based on the distance from the snake head to the target point; reward R for reaching the target collision By judging whether the snake head has successfully reached the target point, a penalty is imposed; the cost penalty R is controlled cost Penalties are imposed based on the magnitude and variability of joint movement.

[0017] The above R forward The calculation formula is: forward =w forward ·v goal , where v goal 、w forwardare the component and weight coefficient of the snake head velocity in the target direction respectively;

[0018] R angle The calculation formula is: angle =cos(θ), where θ is the angle between the snake's moving direction and the target direction;

[0019] R distance The calculation formula is: Among them, Δd represents the change in the distance from the snake head to the target;

[0020] R target The calculation formula is: Among them, d represents the distance from the current snake head to the target point, d threshold is the distance threshold, w 1 is the weight coefficient;

[0021] R collision The calculation formula is: Among them, C collision is a constant;

[0022] R cost Indicates the resources consumed by the robot to perform actions. The calculation formula is: R cost =w 2 ·∑ i |action i | 2 , where w 2 is the weight coefficient of control cost, action i is the size of the ith action.

[0023] The input layer of the action network described in the above step 4 receives state information from the environment, including joint angles, joint speeds, snake head positions and obstacle positions; the hidden layer includes two layers, each layer includes 256 neurons, and the activation function uses ReLU; the output layer outputs the control signal of each joint, and uses the tanh activation function to map the output to the control range; and log_std_parameter is used to control the exploratory nature of the action, and the action is sampled by Gaussian distribution to enhance the robustness of the strategy; the value network is used to evaluate the value function of the current state and assist the strategy network in optimizing its action selection strategy. Its input layer receives state information from the environment, including joint angles, joint speeds, snake head positions and obstacle positions, and the output layer outputs the value estimate of the current state.

[0024] In each training cycle of step 5 above, the snake robot interacts with the environment and collects state s t 、Action a t , Reward R tThe data is stored in the memory bank. The reinforcement learning algorithm uses the data in the memory bank to calculate the clipping objective function and the value loss function. CLIP (θ) updates the parameters θ of the policy network so that the robot can choose better actions to maximize the cumulative reward. At the same time, the value network minimizes the value loss function L VF (φ), updates the value parameter φ to accurately estimate the value of each state and provide effective guidance for the policy network. In the snake robot control task, the total loss function L(θ,φ) is used to ensure the effective training of the policy network and the value network, and the efficiency and stability of its movement are ensured by stabilizing the policy update.

[0025] The above clipping objective function is as follows:

[0026]

[0027] The value loss function is as follows:

[0028]

[0029] The total loss function is as follows:

[0030] L(θ,φ)=L CLIP (θ)-c 1 L VF (φ)+c 2 S[π θ ](s t )

[0031] in,

[0032] represents the average value of all sampled time steps t;

[0033] is the advantage function;

[0034] Clip(r t (θ),1-∈,1+∈) represents the strategy ratio r t (θ) is limited to the range [1-∈, 1+∈];

[0035] ∈ is a constant;

[0036] π θ (a t |s t ) is the current policy network in a given state s t Next select action a t The probability density function of

[0037] For the old policy network in a given state s tNext select action a t The probability density function of

[0038] V φ (s t ) is in state s t Estimation of the value function under ;

[0039] is the target value at time step t;

[0040] S[π θ ](s t ) is in state s t Next strategy π θ Entropy of

[0041] r t (θ) is the strategy ratio of t rounds of training;

[0042] c 1 , c 2 is a hyperparameter.

[0043] Snake robots need to move smoothly in complex environments to avoid drastic changes in motion caused by excessive policy updates. The clipping objective function ensures the smoothness and continuity of the robot's motion by limiting the amplitude of policy updates.

[0044] The present invention has the following beneficial effects:

[0045] The present invention does not rely on visual information, but is mainly controlled based on non-visual information such as joint angles, joint speeds, snake head positions and obstacle positions. It does not rely on cameras or other visual sensors, which significantly reduces the complexity and cost of the system and improves its adaptability in visually constrained environments. It enables the robot to adapt to different environments and challenges, and improves its flexibility and success rate in specific task scenarios.

[0046] The present invention guides the snake robot to optimize its motion strategy by designing a diversified reward function, achieving an efficient forward speed and a stable motion posture, ensuring that the robot can move smoothly in a complex environment and avoiding collisions or loss of control due to unstable motion. At the same time, it can effectively identify and avoid obstacles in the environment, optimize the path planning of the snake robot through low and narrow passages, and improve the robot's navigation ability and task completion rate in a complex environment.

[0047] During the training process, the present invention improves the generalization ability of the model by introducing diverse simulation environment configurations, such as obstacles and low passages of different positions and sizes, so that the trained control strategy can adapt to a variety of different complex environments, thereby enhancing the practicality and flexibility of the system.

[0048] The present invention provides an efficient and adaptable control method by combining a reinforcement learning algorithm with the autonomous control of a snake-like robot. Through autonomous learning, efficient and stable motion control of the snake-like robot in a complex environment is achieved, especially through a low and narrow space such as a hole, from one end to the other without the need for visual information. This significantly improves the movement ability of the snake-like robot in a complex environment, especially through a low and narrow space such as a hole without the need for visual information.

[0049] The core purpose of the objective function clipping of the present invention is to control the change range of the strategy and prevent drastic changes when the strategy is updated, thereby ensuring the stability and convergence of the training. Different from the traditional clipping method, the present invention introduces a dynamic adjustment mechanism, combines the performance and convergence of the current training stage, and dynamically adjusts the clipping ratio ∈, thereby better controlling the range of strategy updates.

[0050] The value loss function aims to optimize the value network's estimate of the state value. In the motion control task in a low and narrow space, the value loss function enhances the value network's evaluation of each action through reward correction. The total loss function adjusts the contribution of each part through weights to ensure the coordination and balance of the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the snake-like robot control method based on reinforcement learning of the present invention;

[0052] Figure 2 This is a schematic diagram of the process of the present invention using a snake-like robot based on reinforcement learning to pass through a low and narrow passage;

[0053] Figure 3 This is a schematic diagram of the process of the reinforcement learning-based snake-like robot passing through a low and narrow passage and reaching a target position. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0056] The present invention provides a control method for a snake-like robot based on reinforcement learning. The method comprises the following steps: constructing a simulation model of the snake-like robot; setting up a training environment for reinforcement learning; obtaining the current state information of the snake-like robot through joint state and position sensors; and selecting actions through a value network to achieve the goal of passing through a low and narrow passage. The reinforcement learning algorithm PPO is applied, and through iterative training, the robot learns a set of parameters for making optimized decisions in different scenarios. This method enables the robot to adapt to different environments and challenges, and improves its flexibility and success rate in specific task scenarios. Figure 1 As shown, the method specifically comprises the following steps:

[0057] Step 1: Build a complex terrain simulation scene including walls, cube obstacles and low passages;

[0058] Step 2: Model the structure, joints and dynamic parameters of the snake-like robot to obtain a simulation model of the snake-like robot;

[0059] Step 3: Define the state space and action space of the snake robot, and design a diversified reward function by comprehensively considering the movement speed of the snake robot's head toward the target direction, the degree of alignment between the movement direction and the target direction, the change in the distance to the target point, the distance to the target point, whether a collision occurs, and the amplitude and change of joint movements;

[0060] Step 4: Use the multi-layer perceptron MLP structure to build the action network and value network;

[0061] Step 5: Based on steps 1 to 4, a reinforcement learning algorithm is used to conduct multiple rounds of simulation training on the snake robot, so that the robot can learn the parameter sets for making optimal decisions in different scenarios, realize the optimal control of the snake robot, and reach the target point.

[0062] The embodiments cover the construction of simulation environment, the design and implementation of reinforcement learning environment, the training and optimization of reinforcement learning model, and the testing and verification of the model, etc., as follows:

[0063] like Figure 2 As shown in the figure, step 1 establishes a simulation scene with complex terrain such as walls, cubic obstacles and low passages. The simulation environment is modeled and simulated using the MuJoCo simulation platform. A plane-type geometry is used as the ground, and walls are set on both sides of the environment to prevent the snake robot from exceeding the predetermined area and guide it through the low passage. Multiple cubic obstacles are placed in the environment to form a narrow passage, ensuring that the snake robot needs to pass through the low passage to continue moving forward. At the same time, the shortest path to the target point requires the narrowest passage, thereby increasing the challenge of the snake robot passing through the low passage.

[0064] According to the above XML configuration file, a simulation model of the snake robot is established, and walls, cube obstacles and low passages are set in the environment. By adjusting the size and position of each geometric body, the simulation environment can simulate the complex terrain in actual applications. In particular, the size and position of the low passage are optimized to ensure that the snake robot can pass smoothly.

[0065] A plurality of light sources are arranged in the simulation environment to ensure good visualization of the robot and obstacles, and to assist in debugging and observing simulation effects.

[0066] Step 2: Model the structure, joints and dynamic parameters of the snake-like robot in detail to obtain a simulation model of the snake-like robot;

[0067] The snake-like robot comprises a snake body connected by a plurality of joints, each joint being controlled by a corresponding actuator. The robot body comprises a snake head, a snake body connected by a plurality of joints and a snake tail.

[0068] Step three designs the state space of the deep reinforcement learning algorithm, including information such as joint angles, joint velocities, snake head position, distance from the target point, and position of obstacles; in the embodiment, the snake robot includes angles (12) and angular velocities (12) of all joints, a total of 24 values, reflecting the current posture and motion state of the snake robot. The position state mainly includes the position coordinates (x, y) of the snake head in the two-dimensional plane, and the distance from the target point, which are used to guide the snake robot to move in the target direction. The obstacle information includes four cubic obstacles, and the height is limited by the top cube.

[0069] The action space is defined as the control signal of each joint. In the embodiment, for each joint of the snake robot, 12 continuous values ​​are defined, each value corresponds to a control signal of a joint, and the range is between [-1, 1]. Through linear mapping, the control signal is converted into the actual control range of the actuator [-7, 7].

[0070] The reward function is the key to guiding the reinforcement learning agent to learn the correct behavior. The present invention designs a diversified reward function, which comprehensively considers factors such as the movement speed of the snake robot in the target direction, the degree of alignment between the movement direction and the target direction, the reward for passing through low passages, the penalty for avoiding obstacles, and the cost of controlling energy consumption, so as to guide the robot to learn the optimal movement strategy; the reward function includes forward reward, angle reward, distance change reward, target distance penalty and goal achievement reward, which encourages the snake robot to move in the target direction, avoid collisions, and pass through low passages smoothly, so as to guide the snake robot to learn a strategy of efficiently passing through low passages and avoiding obstacles.

[0071] The current calculation method of the diversified reward value Rt is as follows:

[0072] R t =R forward +R angle +R distance +R target +R collision +R cost

[0073] The forward reward is given based on the speed of the snake head moving towards the target direction, mainly by calculating the component v of the snake head speed in the target direction. goal , multiplied by the weight coefficient w forward The specific formula is as follows:

[0074] R forward =w forward ·v goal

[0075] The angle reward is the degree of alignment between the movement direction of the snake head and the target direction. By calculating the angle θ between the actual movement direction of the snake head and the target direction, taking the cosine value and multiplying it by the weight coefficient, the closer the movement direction is to the target direction, the greater the reward. When the movement direction of the snake head is consistent with the target direction, the reward is the largest (cos(0)=1); when the directions are opposite, the reward is the smallest (cos(π)=-1). The specific formula is as follows:

[0076] R angle =cos(θ)

[0077] The distance change reward is mainly based on the change in the distance from the snake head to the target point. If the current distance is reduced compared to the previous moment, a reward is given; if it increases, a penalty is given. Δd represents the change in the distance to the target. The specific formula is as follows:

[0078]

[0079] Δd=d previous -d current

[0080] The target distance penalty is based on the distance from the snake head to the target point, which is used to encourage the robot to approach the target. When the distance exceeds the set threshold of 0.1 meters, a negative reward proportional to the distance is given, and the maximum penalty value is set to -1.0. d represents the current distance from the snake head to the target point, d threshold is the distance threshold, which is set to 0.1 here, representing the radius of the green target ball in the simulation. The specific formula is as follows:

[0081]

[0082] Among them, w 1 is the weight coefficient.

[0083] By checking the contact with the obstacle, it is determined whether a collision occurs. If a collision occurs, an additional negative reward self._collision_penalty is given. The channel range is defined according to the pos and size of low_passage in XML. By checking whether the snake head position is within the predetermined range of the low channel, it is determined whether a collision occurs. The specific formula is as follows:

[0084]

[0085] Among them, C collision is a constant.

[0086] Penalties are imposed based on the amplitude and variation of joint motions to encourage energy conservation and stable control. The method is to calculate the control cost penalty R cost Represents the resources consumed by the robot to perform actions. Large changes in joint angles require more energy, while small changes in angles consume less energy. The calculation formula is: R cost =w 2 ·∑ i |action i | 2 , where w 2 is the weight coefficient of control cost, action i is the size of the ith action.

[0087] The above rewards and penalties are weighted and superimposed to form the final reward signal.

[0088] Step 4: Use the multi-layer perceptron MLP structure to build the action network and value network;

[0089] In terms of network structure design, a multi-layer perceptron MLP structure is used to establish an action neural network Actor. The design of the action network aims to generate appropriate control signals according to the current environmental state to guide the snake robot to perform effective motion actions, especially to maintain efficiency and stability when passing through low and narrow spaces. Its input layer receives state information from the environment, including joint angles, joint speeds, snake head positions, and obstacle positions, etc. The total input dimension is 40. The hidden layer contains two layers, each containing 256 neurons. The activation function uses ReLU to enhance the nonlinear expression ability of the network and ensure that the network can effectively capture the relationship between complex states and actions. The output layer outputs the control signal of each joint, and uses the tanh activation function to map the output to the control range. The tanh activation function limits the output action to the range of [-1,1] to meet the control requirements of each joint of the snake robot. Then log_std_parameter is used to control the exploratory nature of the action, and the action is sampled by Gaussian distribution to enhance the robustness of the strategy.

[0090] The value neural network Critic is established. The value network Critic is used to evaluate the value function of the current state and assist the policy network in optimizing its action selection strategy. The design of the value network also adopts the multi-layer perceptron MLP structure to ensure accurate value estimation. Its input layer accepts the same state information as the policy network, and the output layer outputs the value estimation of the current state.

[0091] Step 5: Training and implementing the reinforcement learning model: The reinforcement learning algorithm is used to train the snake robot. Through multiple rounds of simulation training, the robot gradually optimizes its motion strategy and can autonomously avoid obstacles and pass through low passages in extremely narrow and complex environments.

[0092] Specifically, the current joint angle, joint speed, snake head position and obstacle position are obtained from the simulation environment to form a state vector. The reinforcement learning agent selects an action based on the current state. The action is converted into a control signal of the actuator through linear mapping and applied to the simulation environment.

[0093] According to the set rendering mode human, the movement state of the snake robot is visualized in real time, which is convenient for observation and debugging.

[0094] By running simulations, we ensure that the positions and dimensions of all obstacles and passages in the environment are set correctly, and the snake robot can move freely in the environment.

[0095] Use the visualization tool to check the interaction between the snake robot and obstacles to ensure that the collision detection function is working properly.

[0096] The performance of the training model is verified through simulation tests and actual applications. The reward function weights, algorithm parameters and model structure are adjusted according to the evaluation results to improve the stability and generalization ability of the control method.

[0097] In order to realize the autonomous control of the snake-like robot in a complex environment, the present invention adopts the Proximal Policy Optimization (PPO) algorithm in the Stable Baselines3 library to train the snake-like robot and optimize its autonomous control ability in a complex environment.

[0098] In the present invention, the PPO algorithm used is based on the policy gradient method, which ensures the stability of the policy update by limiting the amplitude of the policy update. The PPO algorithm updates the policy parameter θ by the following clipping objective function, updates the value parameter φ by minimizing the following value loss function, and combines it with the control of the snake robot through the following formula to achieve efficient and stable autonomous navigation and obstacle avoidance capabilities. In the snake robot control task, the total loss function L(θ,φ) is used to ensure the effective training of the policy network and the value network, and the efficiency and stability of its movement are ensured by stabilizing the policy update.

[0099] The specific formula of the strategy ratio is as follows, where the numerator is the action probability density function output by the current strategy network, and the denominator is the action probability density function of the old strategy, which is used to measure the difference between the new and old strategies in the current action selection.

[0100]

[0101] The specific formula of the clipping objective function is as follows:

[0102]

[0103] The specific formula of the value loss function is as follows:

[0104]

[0105] The specific formula of the total loss function is as follows:

[0106] L(θ,φ)=L CLIP (θ)-c 1 L VF (φ)+c 2 S[π θ ](s t )

[0107] in It means taking the average of all sampled time steps t and performing expected calculation on the entire batch of data to ensure the stability and consistency of the strategy update.

[0108] is the advantage function, calculated by the generalized advantage estimation (GAE) method, representing action a t In status t The advantage of the next state relative to the current state. The calculation formula is as follows:

[0109] A t =R t -V φ (s t )

[0110] Clip is to limit the strategy ratio rt(θ) to the range of [1-∈, 1+∈], so that the strategy is updated in a favorable direction and the advantage function is used. To guide optimization.

[0111] ∈ is a small constant that controls the extent of clipping.

[0112] π θ (a t |s t ) is the current policy network in a given state s t Next select action at The probability density function of .

[0113] For the old policy network in a given state s t Next select action a t The probability density function of .

[0114] V φ (s t ) is in state s t The value function estimate under .

[0115] is the target value at time step t, used to train the value network.

[0116] S[π θ ](s t ) is in state s t Next strategy π θ The entropy is used to measure the strategy in state s t The randomness of the following.

[0117] c 1 , c 2 are hyperparameters used to weigh the value loss and entropy regularization term respectively.

[0118] In each training cycle, the snake robot interacts with the environment and collects the state s t 、Action a t , reward Rt and other data, and store them in the memory bank.

[0119] Through the PPO algorithm, the loss function is calculated using the above formula, and the parameters θ of the policy network are updated, so that the robot can choose a better action to maximize the cumulative reward. At the same time, the value network updates the parameter φ by minimizing the value loss function to accurately estimate the value of each state, providing effective guidance for the policy network.

[0120] The learning rate is set to 3e-4 to balance the learning speed and stability, ensuring that the model can learn effectively without causing training instability due to excessive learning rate. The discount factor is set to 0.99 to consider long-term rewards, emphasize long-term rewards, and promote long-term planning of snake robots in complex environments. The number of policy updates is set to 2048 to ensure that enough samples are collected before each policy update to improve training efficiency.

[0121] Use the check_env function to ensure that the environment meets the OpenAI Gym interface requirements. Start the reinforcement learning training process by calling the learn method and setting the total number of training steps to 500,0000.

[0122] Use the PPO.load method to load the previously saved training model. Before the test starts, reset the simulation environment, initialize the state of the snake robot, use the trained model to predict the action, and apply the predicted action to the simulation environment. Use real-time visualization to observe whether the snake robot can smoothly pass through the low channel and avoid all obstacles.

[0123] After the training is completed, the snake robot is tested using the trained model to verify its ability to pass through low passages in complex environments.

[0124] Depend on Figure 2 It can be clearly seen that in the simulation environment, the method proposed in the present invention successfully guides the snake-like robot through a low passage.

[0125] Depend on Figure 3 It can be clearly seen that in the simulation environment, the method proposed in the present invention successfully guides the snake-like robot through the low passage and then reaches the target position.

[0126] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0127] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A snake-like robot control method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Build a complex terrain simulation scene including walls, cube obstacles and low passages; Step 2: Model the structure, joints and dynamic parameters of the snake-like robot to obtain a simulation model of the snake-like robot; Step 3: Define the state space and action space of the snake robot, and design a diversified reward function by comprehensively considering the movement speed of the snake robot's head toward the target direction, the degree of alignment between the movement direction and the target direction, the change in the distance to the target point, the distance to the target point, whether a collision occurs, and the amplitude and change of joint movements; Step 4: Use the multi-layer perceptron MLP structure to build the action network and value network; Step 5: Based on steps 1 to 4, a reinforcement learning algorithm is used to conduct multiple rounds of simulation training on the snake robot, so that the robot can learn the parameter sets for making optimal decisions in different scenarios, realize the optimal control of the snake robot, and reach the target point.

2. A snake-like robot control method based on reinforcement learning according to claim 1, characterized in that: In step 1, multiple light sources are set in the simulation environment to ensure the visualization of the robot and obstacles, and to assist in debugging and observing the simulation effect.

3. The control method of a snake-like robot based on reinforcement learning according to claim 1, characterized in that: The state space of step 3 includes the joint angles, joint velocities, snake head position, distance between the snake robot and the target point, and the position of obstacles of the snake robot; the action space is the control signal of each joint of the snake robot.

4. The control method of a snake-like robot based on reinforcement learning according to claim 1, characterized in that: The diversified reward function described in step 3 is: R t =R forward +R angle +R distance +R target +R collision +R cost , where the forward reward R forward Rewards are given based on the speed at which the snake head moves towards the target direction; angle rewards R angle Rewards are given based on the alignment between the movement direction of the reward snake head and the target direction; distance change reward R distance Reward or penalty is given based on the change in the distance from the snake head to the target point; target distance penalty R target Penalty is given based on the distance from the snake head to the target point; reward R for reaching the target collision By judging whether the snake head has successfully reached the target point, a penalty is imposed; the cost penalty R is controlled cost Penalties are imposed based on the magnitude and variability of joint movement.

5. The control method of a snake-like robot based on reinforcement learning according to claim 1, characterized in that: The R forward The calculation formula is: forward =w forward ·v goal , where v goal 、w forward are the component and weight coefficient of the snake head velocity in the target direction respectively; R angle The calculation formula is: angle =cos(θ), where θ is the angle between the snake's moving direction and the target direction; R distance The calculation formula is: Among them, Δd represents the change in the distance from the snake head to the target; R target The calculation formula is: Among them, d represents the distance from the current snake head to the target point, d threshold is the distance threshold, w1 is the weight coefficient; R collision The calculation formula is: Among them, C collision is a constant; R cost Indicates the resources consumed by the robot to perform actions. The calculation formula is: R cost =w2·∑action i 2 , where w2 is the weight coefficient of the control cost, action i is the size of the ith action.

6. The control method of a snake-like robot based on reinforcement learning according to claim 1, characterized in that: In step 4, the input layer of the action network receives state information from the environment, including joint angles, joint speeds, snake head positions and obstacle positions; the hidden layer contains two layers, each containing 256 neurons, and the activation function uses ReLU; the output layer outputs the control signal of each joint, and uses the tanh activation function to map the output to the control range; and log_std_parameter is used to control the exploratory nature of the action, and the action is sampled by Gaussian distribution to enhance the robustness of the strategy; the value network is used to evaluate the value function of the current state and assist the strategy network in optimizing its action selection strategy. Its input layer receives state information from the environment, including joint angles, joint speeds, snake head positions and obstacle positions, and the output layer outputs the value estimate of the current state.

7. The control method of a snake-like robot based on reinforcement learning according to claim 1, characterized in that: In each training cycle of step 5, the snake robot interacts with the environment and collects state s t 、Action a t , Reward R t The data is stored in the memory bank. The reinforcement learning algorithm uses the data in the memory bank to calculate the clipping objective function and the value loss function. CLIP (γ) Update the parameters θ of the policy network so that the robot can choose better actions to maximize the cumulative reward. At the same time, the value network minimizes the value loss function L VF (φ), updates the value parameter φ to accurately estimate the value of each state and provide effective guidance for the policy network. In the snake robot control task, the total loss function L(θ,φ) is used to ensure the effective training of the policy network and the value network, and the efficiency and stability of its movement are ensured by stabilizing the policy update.

8. The control method of a snake-like robot based on reinforcement learning according to claim 7, characterized in that: The clipping objective function is as follows: The value loss function is as follows: The total loss function is as follows: L(θ,φ)=L CLIP (θ)-c1L VF (φ)+c2S[π θ ](s t ) in, It means taking the average value of all sampled time steps t; is the advantage function, representing action a t In status t The advantage of the next state relative to the current state is calculated as follows: A t =R t -V φ (s t ) Clip(r t (θ),1-∈,1+∈) represents the strategy ratio r t (θ) is limited to the range [1-∈, 1+∈]; ∈ is a constant; π θ (a t |s t ) is the current policy network in a given state s t Next select action a t The probability density function of For the old policy network in a given state s t Next select action a t The probability density function of V φ (s t ) is in state s t Estimation of the value function under ; is the target value at time step t; S[π θ ](s t ) is in state s t Next strategy π θ Entropy of r t (θ) is the strategy ratio of t rounds of training; c1, c2 are hyper parameters.

Images