A spiral driving snake robot trajectory tracking method based on reinforcement learning
By using a reinforcement learning-based approach and employing optical motion capture and a policy neural network model to control the rotation of the helical joint, the problem of poor trajectory tracking control in helical-driven snake robots was solved, achieving accurate trajectory tracking and improving the ground movement capability of the helical-driven snake robot.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANKAI UNIV
- Filing Date
- 2025-04-08
- Publication Date
- 2026-05-15
AI Technical Summary
Existing trajectory tracking methods are not effective for controlling helical-driven snake robots, and there is a lack of research on helical-driven snake robots, resulting in imperfect ground motion functions.
A reinforcement learning-based approach is adopted to obtain the current position state of a helical-driven snake robot through an optical motion capture system. By using a policy neural network model and model predictive control, the execution actions of the helical joints are obtained, thereby achieving precise trajectory tracking control.
It achieves precise trajectory tracking for a helical-driven snake robot, providing the basis for its ground movement and laying the foundation for subsequent functional development, thus possessing high practical application value.
Smart Images
Figure CN120244961B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of trajectory tracking control technology, and in particular to a trajectory tracking method for a helical-driven snake robot based on reinforcement learning. Background Technology
[0002] Snake-like robots are a common type of biomimetic robot that mimics the body structure and movement of snakes to achieve serpentine locomotion. Characterized by their elongated bodies and high redundancy, snake-like robots are typically used in low-ceilinged, confined environments, such as rubble search and rescue, pipeline inspection, and field reconnaissance. Their ground-based locomotion capabilities are fundamental to these advanced biomimetic functions. They can approach specific environments or terrains remotely or via navigation from the ground before performing these functions. Precise trajectory tracking control is essential for achieving remote control or navigation.
[0003] Common snake-like robot building blocks include yaw joints, pitch joints, wheel joints, and telescopic joints. Snake-like robots constructed from these joints typically possess fixed gaits, and their trajectory tracking control is based on these gaits. The helical joint is a novel type of driven joint. Snake-like robots driven by helical joints currently lack fixed gaits, making traditional methods unsuitable. Furthermore, research on helical-driven snake-like robots is limited, and according to surveys, there is a lack of research on trajectory tracking control for helical-driven snake-like robots. Therefore, a reinforcement learning-based trajectory tracking method for helical-driven snake-like robots is urgently needed to improve their ground motion capabilities. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a reinforcement learning-based trajectory tracking method for helical-driven snake robots, which can precisely control the trajectory tracking of helical-driven snake robots.
[0005] A reinforcement learning-based method for tracking the trajectory of a spiral-driven snake robot includes the following steps:
[0006] Step S1: Obtain the current position state of the spiral-driven snake robot through an optical motion capture system. The current position state includes the X coordinate, Y coordinate, and yaw angle of the spiral-driven snake robot's location.
[0007] Step S2: Based on the current position state and the current desired target point of the spiral-driven snake robot, the execution actions of the spiral joints of the spiral-driven snake robot are obtained through a policy neural network model and model predictive control; the execution actions are used to control the rotation speed and rotation direction of the spiral joints.
[0008] Step S3: The spiral-driven snake robot performs motion control according to the actions of the spiral joints.
[0009] As one implementation of this application, the method for training the policy neural network model is as follows:
[0010] Step S21: Construct a state transition model for the helical-driven snake robot; the state transition model is used to predict the position state of the helical-driven snake robot after it performs an action;
[0011] Step S22: Constructing an intelligent agent: The intelligent agent includes the policy neural network model and the behavior value function neural network model; the behavior value function neural network model is used to output a quality evaluation of the currently executed action based on the position information of the spiral-driven snake robot;
[0012] Step S23: Initialize the position state of the helical-driven snake robot and set the desired motion trajectory; the desired motion trajectory consists of multiple desired target points;
[0013] Step S24: The intelligent agent outputs an action to be executed based on the current position state of the spiral-driven snake robot, changing the rotation speed and direction of the spiral joint;
[0014] Step S25: The state transition model outputs the position state of the helical-driven snake robot at the next moment based on the current position state of the helical joint, the rotation speed and rotation direction of the helical joint, and calculates and saves the reward function; the reward function is the negative value of the Euclidean distance from the position of the helical-driven snake robot to the desired target point; after reaching the destination position, the reward is calculated;
[0015] Step S26: Training the agent: Based on the position state, executed action, and reward function obtained in step S25, update the behavior value function neural network model using the temporal difference method to obtain the behavior value function; update the policy neural network model based on the negative value of the behavior value function using a greedy policy;
[0016] Step S27: Repeat steps S23-S26 until the preset number of iterations is reached, then end the training.
[0017] As one implementation of this application, the state transition model is a radial basis function neural network model.
[0018] As one implementation of this application, the state transition model uses the mean squared error as the loss function.
[0019] As one implementation of this application, the policy neural network model is constructed using a multilayer perceptron.
[0020] As one implementation of this application, the behavioral value function neural network model consists of two network models constructed using a multilayer perceptron. Each behavioral value function neural network model has a three-layer fully connected network structure, and each fully connected layer is a linear layer.
[0021] As one implementation of this application, the execution action output in step S24 remains constant within the current step.
[0022] As one implementation of this application, in step S26, the behavior value function neural network model is updated using the temporal difference method to obtain the behavior value function using the following formula:
[0023] ;
[0024] in, This represents the target behavior value function. and These represent the behavior-valued function neural network model, respectively. and behavioral value function neural network model The output, Indicates the current position status. Indicates the currently executing action. Indicates the state Execute action at time The reward Indicates the discount factor. Indicates the state Execute action The next state after that, Indicates the state The action to be performed at the location.
[0025] As one implementation of this application, the spiral-driven snake robot is composed of alternating spiral joints and yaw joints.
[0026] As one implementation of this application, the structure of the policy neural network model is a three-layer fully connected layer.
[0027] The embodiments of the present invention have the following technical effects:
[0028] This application provides a reinforcement learning-based trajectory tracking method for a helical-driven snake robot. The method captures the current position state of the helical-driven snake robot through optical motion, including the X-coordinate, Y-coordinate, and yaw angle of the robot's current location. Based on the current position state and the desired target point, a policy neural network model and model predictive control are used to obtain the execution actions of the helical joints. These execution actions control the rotational speed and direction of the helical joints. The helical-driven snake robot performs motion control according to these execution actions. By obtaining the execution actions of the helical joints based on the current position state and the desired target point, the helical-driven snake robot moves along a predetermined trajectory, ensuring that it can track the predetermined trajectory and achieving precise trajectory tracking control. This provides a foundation for subsequent functional development based on the helical-driven snake robot and has high practical application value. Attached Figure Description
[0029] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0030] Figure 1 A schematic diagram illustrating the steps of a reinforcement learning-based spiral-driven snake robot trajectory tracking method provided in an embodiment of the present invention;
[0031] Figure 2 A flowchart illustrating a reinforcement learning-based trajectory tracking method for a spiral-driven snake robot, provided in an embodiment of the present invention;
[0032] Figure 3 This is a physical image of the spiral-driven snake robot constructed according to the present invention;
[0033] Figure 4 Train the loss function graph for the state transition model;
[0034] Figure 5 Train the loss function graph for the behavior-valued neural network model;
[0035] Figure 6 For cumulative return charts;
[0036] Figure 7This is a diagram showing the motion trajectory of a spiral-driven snake robot. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0038] To address the issue that existing trajectory tracking methods are ineffective for controlling helical-driven snake robots due to their different actuation methods, this application provides a reinforcement learning-based trajectory tracking method for helical-driven snake robots. The process is as follows: Figure 2 .
[0039] A reinforcement learning-based method for tracking the trajectory of a spiral-driven snake robot, consisting of... Figure 1 It can be seen that the specific steps include the following:
[0040] Step S1: Obtain the current position state of the spiral-driven snake robot through an optical motion capture system. The current position state includes the X coordinate, Y coordinate, and yaw angle of the spiral-driven snake robot's location.
[0041] Step S2: Based on the current position state and the current desired target point of the spiral-driven snake robot, the execution actions of the spiral joints of the spiral-driven snake robot are obtained through a policy neural network model and model predictive control; the execution actions are used to control the rotation speed and rotation direction of the spiral joints.
[0042] For example, the target point can be represented in coordinate form.
[0043] As one implementation of this application, the policy neural network model is constructed using a multilayer perceptron.
[0044] As one implementation of this application, the structure of the policy neural network model is a three-layer fully connected layer. For example, the number of neurons in the first and second layers is set to 7, and the number of neurons in the third layer is set to 3.
[0045] As one implementation of this application, the method for training the policy neural network model is as follows:
[0046] Step S21: Construct a state transition model for the helical-driven snake robot; the state transition model is used to predict the position state of the helical-driven snake robot after it performs an action.
[0047] For example, a two-dimensional helical-driven snake robot composed of alternating helical and yaw joints is constructed, see... Figure 3 The constructed snake-like robot has seven joints, a total length of 0.9m, a diameter of 0.09m, and a weight of approximately 2.5kg. Each joint is powered by an independent battery, and the head contains a chip and a wireless transmitter / receiver module. The yaw joint uses servos with parallel axes pointing in the same direction, while the helical joint uses DC black-plated motors. The snake's head receives angles, helical speeds, and directions calculated by the PC and then transmits these calculations to each joint via an RS485 bus.
[0048] A state transition model for a helical-driven snake robot is constructed. The rotational speed command for each helical joint is set to [200, 220, 240, 260, 280, 300, 320, 340, 360, 380, 400]. The speeds of the four helical joints are combined according to these command values and then issued. The angles issued for the three yaw joints are 30°, 150°, and 30°, respectively, and remain constant. Each speed combination runs for 2 seconds, with a sampling time of 0.02 seconds. The position information of the constructed helical-driven snake robot under different speed combinations is obtained using an optical motion capture system.
[0049] As one implementation of this application, the state transition model is a radial basis function (RBF) neural network model.
[0050] As one implementation of this application, the state transition model uses mean squared error as the loss function; for example, the number of neurons is set to 100.
[0051] Step S22: Constructing an intelligent agent: The intelligent agent includes the policy neural network model and the behavior value function neural network model; the behavior value function neural network model is used to output a quality evaluation of the currently executed action based on the position information of the spiral-driven snake robot;
[0052] In one implementation of this application, the behavior value function neural network model comprises two network models constructed using a multilayer perceptron. Each behavior value function neural network model has a three-layer fully connected network structure, with each fully connected layer being a linear layer. The input to the behavior value function neural network model is the current state of the spiral-driven snake robot and the next action to be executed. The output is the cumulative reward obtained after executing the input action in the current state. The larger the cumulative reward, the better the current action is executed. For the same state and action input, the smaller value of the outputs of the two behavior value function neural network models is selected as the final output, thus avoiding the overestimation problem that exists when using a single behavior value function model. The action evaluation given by the behavior value function model serves as the basis for the action output of the policy neural network model. For example, the number of neurons in the first and second fully connected networks is set to 7, and the number of neurons in the third fully connected network is set to 1. This configuration of the behavior value function neural network model can accurately evaluate the quality of the current action and has high computational efficiency.
[0053] Step S23: Initialize the position state of the spiral-driven snake robot and set the desired motion trajectory; the desired motion trajectory consists of multiple desired target points.
[0054] For example, the desired trajectory is set as a straight line from the starting point coordinates (0.25, -1.15) to the ending point coordinates (-0.2, 0.17).
[0055] Step S24: The intelligent agent outputs an action to be executed based on the current position state of the spiral-driven snake robot, changing the rotation speed and direction of the spiral joint;
[0056] As one implementation of this application, the execution action output in step S24 remains constant within the current step.
[0057] Step S25: The state transition model outputs the position state of the helical-driven snake robot at the next moment based on the current position state of the helical joint, the rotation speed and rotation direction of the helical joint, and calculates and saves the reward function; the reward function is the negative value of the Euclidean distance from the position of the helical-driven snake robot to the desired target point; after reaching the destination position, the reward is calculated;
[0058] The preset trajectory consists of multiple desired target points. By tracking each desired target point in an orderly manner, the spiral-driven snake robot can track the preset trajectory, achieving precise control of the spiral-driven snake robot's trajectory tracking. This provides a foundation for subsequent functional development based on the spiral-driven snake robot and has high practical application value.
[0059] Starting from an initial state, the agent randomly generates actions, interacts with the state transition model, obtains the state at the next moment, calculates the reward function, and then continues to generate actions, repeating this cycle until the endpoint or maximum number of steps is reached, and the trajectory data is saved. The number of rounds is set to 5, and the maximum number of steps in a single round is 200.
[0060] Step S26: Training the agent: Based on the position state, executed action, and reward function obtained in step S25, update the behavior value function neural network model using the temporal difference method to obtain the behavior value function; update the policy neural network model based on the negative value of the behavior value function using a greedy policy;
[0061] As one implementation of this application, in step S26, the behavior value function neural network model is updated using the temporal difference method to obtain the behavior value function using the following formula:
[0062] ;
[0063] in, This represents the target behavior value function. and These represent the behavior-valued function neural network model, respectively. and behavioral value function neural network model The output, Indicates the current position status. Indicates the currently executing action. Indicates the state Execute action at time The reward Indicates the discount factor. Indicates the state Execute action The next state after that, Indicates the state The action to be performed at the location.
[0064] Step S27: Repeat steps S23-S26 until the preset number of iterations is reached, then end the training.
[0065] In one implementation of this application, the preset number of iterations is 500. A number of iterations of 500 allows for a balance between computational efficiency and accuracy.
[0066] In the new iteration, the agent combines a policy neural network model with model predictive control (MPC algorithm) to output the action to be executed.
[0067] Specifically, the agent generates 500 action sequences of length 5, interacts with the state transition model one by one, predicts future states, and calculates discounted rewards. The first action in the action sequence with the largest discounted reward is selected as the optimal action for the current state and output. Figure 4 Train the loss function graph for the state transition model. Figure 5 To train the loss function graph for the behavior-valued neural network model, Figure 6 For the cumulative return chart, Figure 7 This is a diagram showing the motion trajectory of a spiral-driven snake robot.
[0068] Step S3: The spiral-driven snake robot performs motion control according to the actions of the spiral joints.
[0069] In this invention, based on the current position and desired target point of the helical-driven snake robot, the execution actions of the helical joints of the helical-driven snake robot are obtained through a strategy neural network model and model predictive control. This enables the helical-driven snake robot to move along the desired preset trajectory, ensuring that the helical-driven snake robot can track the preset trajectory and achieve precise control of the trajectory tracking of the helical-driven snake robot. This provides a foundation for subsequent functional development based on the helical-driven snake robot and has high practical application value.
[0070] It should be noted that the terminology used in this invention is for describing specific embodiments only and is not intended to limit the scope of this application. As shown in this specification, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may include the plural. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element.
[0071] It should also be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," "linked," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood according to the specific circumstances.
[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.
Claims
1. A method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Obtain the current position state of the spiral-driven snake robot through an optical motion capture system. The current position state includes the X coordinate, Y coordinate, and yaw angle of the spiral-driven snake robot's location. Step S2: Based on the current position state and the current desired target point of the spiral-driven snake robot, the execution actions of the spiral joints of the spiral-driven snake robot are obtained through a policy neural network model and model predictive control; the execution actions are used to control the rotation speed and rotation direction of the spiral joints. The method for training the policy neural network model is as follows: Step S21: Construct a state transition model for the helical-driven snake robot; the state transition model is used to predict the position state of the helical-driven snake robot after it performs an action; Step S22: Constructing an intelligent agent: The intelligent agent includes the policy neural network model and the behavior value function neural network model; the behavior value function neural network model is used to output a quality evaluation of the currently executed action based on the position information of the spiral-driven snake robot; Step S23: Initialize the position state of the helical-driven snake robot and set the desired motion trajectory; the desired motion trajectory consists of multiple desired target points; Step S24: The intelligent agent outputs an action to be executed based on the current position state of the spiral-driven snake robot, changing the rotation speed and direction of the spiral joint; Step S25: The state transition model outputs the position state of the helical-driven snake robot at the next moment based on the current position state of the helical joint, the rotation speed and rotation direction of the helical joint, and calculates and saves the reward function; the reward function is the negative value of the Euclidean distance from the position of the helical-driven snake robot to the desired target point; after reaching the destination position, the reward is calculated; Step S26: Training the agent: Based on the position state, executed action, and reward function obtained in step S25, update the behavior value function neural network model using the temporal difference method to obtain the behavior value function; update the policy neural network model based on the negative value of the behavior value function using a greedy policy; Step S27: Repeat steps S23-S26 until the preset number of iterations is reached, then end the training; Step S3: The spiral-driven snake robot performs motion control according to the actions of the spiral joints.
2. The method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning according to claim 1, characterized in that, The state transition model is a radial basis function neural network model.
3. The method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning according to claim 2, characterized in that, The state transition model is trained using mean squared error as the loss function.
4. The method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning according to claim 1, characterized in that, The proposed strategy neural network model is constructed using a multilayer perceptron.
5. The method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning according to claim 1, characterized in that, The behavioral value function neural network model consists of two network models constructed using a multilayer perceptron. Each behavioral value function neural network model has a three-layer fully connected network structure, and each fully connected layer is a linear layer.
6. The method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning according to claim 1, characterized in that, The execution action output in step S24 remains constant within the current step.
7. The method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning according to claim 1, characterized in that, In step S26, the behavior value function neural network model is updated using the temporal difference method, and the behavior value function is obtained using the following formula: ； in, This represents the target behavior value function. and These represent the behavior-valued function neural network model, respectively. and behavioral value function neural network model The output, Indicates the current position status. Indicates the currently executing action. Indicates the state Execute action at time The reward Indicates the discount factor. Indicates the state Execute action The next state after that, Indicates the state The action to be performed at the location.
8. The method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning according to claim 1, characterized in that, The spiral-driven snake robot is composed of alternating spiral joints and yaw joints.
9. A method for tracking the trajectory of a spiral-driven snake robot based on reinforcement learning according to claim 4, characterized in that, The structure of the policy neural network model is a three-layer fully connected layer.