Underwater snake-like robot single-propeller cooperative control method based on reinforcement learning

Abstract

The present application relates to a method for cooperative control of a single propeller of an underwater snake robot based on reinforcement learning, comprising establishing a structural model of the underwater snake robot, constructing a motion control and decision model of the underwater snake robot, defining state space and action space, and integrating the multi-joint wave propulsion of the snake body and the thrust output of the tail propeller from the central pattern generator into the control decision framework, designing a composite reward function with path tracking error, heading deviation and propulsion efficiency as targets, training the motion control and decision model of the underwater snake robot based on a reinforcement learning algorithm, generating an optimal underwater snake robot cooperative control strategy to execute a multi-waypoint path tracking task of the underwater snake robot, so that the cooperative control strategy of the joint waveform parameters and the propeller thrust is autonomously learned without relying on an accurate hydrodynamic model, thereby realizing stable and high-precision path tracking of a hyper-redundant underwater snake robot under multi-path constraints including turning sections.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater robot control, and particularly relates to a single-propeller cooperative control method for an underwater snake robot based on reinforcement learning. BACKGROUND

[0002] The underwater snake robot has a wide application prospect in underwater detection, pipeline inspection, ecological monitoring and narrow space operation due to its high degree of freedom, strong maneuverability and good adaptability to complex underwater environment. Compared with the traditional propeller underwater robot, the snake robot realizes the wave propulsion mode similar to biological fish or snake through the multi-joint series structure, and has better passability and anti-interference ability in complex environment.

[0003] The existing underwater snake robot mainly adopts the following two types of propulsion and control methods: one type is a control method based on bionic lateral wave propulsion, which usually generates periodic joint swing through a central pattern generator (CPG) to interact with the water body to generate propulsion force; the other type is to set a tail propeller at the end of the snake body, and the tail propeller provides the main axial thrust, and the attitude adjustment and turning control are realized through the pitch and roll of the joint. The above methods can realize the basic underwater motion function in the regular environment and low interference condition, but still have obvious deficiencies in the complex hydrodynamic environment and high maneuvering task scene.

[0004] Firstly, when the snake robot moves, the overall snake body usually presents a clear S-shaped form, at this time the local coordinate system where the tail propeller is located will be deflected with the snake body posture, resulting in that the actual thrust direction of the propeller is inconsistent with the expected forward direction of the robot. Since the thrust output by the propeller cannot always be aligned with the global forward direction, the effective propulsion component is significantly reduced, thereby causing the propulsion efficiency to decrease, and even causing the problem of motion instability under some working conditions.

[0005] Secondly, the thrust direction of the tail propeller in the prior art is usually fixed and cannot be actively adjusted according to the real-time posture change of the snake robot. In actual control, yaw and pitch mainly rely on artificial adjustment of the snake joint waveform parameters to indirectly realize, and the process highly depends on experience, the control coupling relationship is complex, and it is difficult to realize accurate compensation in a dynamic environment, which limits the application ability of the snake robot in complex trajectory tracking and fine maneuvering tasks.

[0006] Thirdly, the traditional control method based on CPG usually adopts fixed or semi-fixed parameter configuration, and the joint swing amplitude, frequency and phase difference are usually set before the task starts. When the water flow speed and direction change, or in the conditions of sharp turns and obstacle avoidance, the snake body motion parameters and the tail thruster thrust command need to be adjusted to maintain the stability of the motion and the efficiency of the propulsion. However, the traditional control method is difficult to solve the optimal control parameters in real time in the absence of an accurate hydrodynamic model, resulting in limited adaptability of the system under complex hydrodynamic disturbance conditions.

[0007] Therefore, how to realize the coordinated control of the tail thruster thrust and the joint waveform considering the overall posture change of the snake body, and enable the control strategy to adjust autonomously in an uncertain hydrodynamic environment, is a key technical problem that needs to be solved in the current field of underwater snake robot control. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides a single-propeller cooperative control method for an underwater snake robot based on reinforcement learning, which solves the problems of reduced propulsion efficiency and unstable motion caused by frequent changes in the direction of the local coordinate system due to the periodic swing of the snake body in the traditional control method based on the tail thruster, and the problem that the traditional control method based on the central pattern generator (CPG) relies on fixed initial parameters and is difficult to adapt to changes in complex hydrodynamic environments. The present application is directed to the motion mechanism characteristics of the underwater snake robot "coupling of snake body wave propulsion and tail thruster jet propulsion in the hydrodynamic environment", and unifies the snake body multi-joint wave propulsion parameters from the CPG and the tail thruster thrust output into the same control decision framework, and takes the underwater dynamics response including buoyancy, fluid resistance and viscous effect as the optimization basis of the control strategy, so that the control strategy can adaptively adjust the propulsion mode under different postures, different fluid resistance and different hydrodynamic disturbance conditions. On this basis, the reinforcement learning method is introduced, and the underwater snake robot and the hydrodynamic environment in which it is located are modeled as a Markov decision process, and the joint swing parameters and the tail thruster thrust are jointly optimized and controlled through the interaction between the agent and the underwater simulation environment containing the hydrodynamic action, so as to realize the multi-waypoint path tracking task in a complex hydrodynamic environment.

[0009] To achieve the above technical purposes, the present application provides the following technical scheme: a single-propeller cooperative control method for an underwater snake robot based on reinforcement learning, comprising the following steps:

[0010] establishing an underwater snake robot structure model; the underwater snake robot structure model comprises:

[0011] a central pattern generator phase update model for describing the central pattern generator phase;

[0012] A joint periodic swing model is used to calculate the swing angle of each joint of the underwater snake robot;

[0013] A tail single-propeller model is used to describe the axial thrust provided by the tail propeller;

[0014] A motion control and decision model of the underwater snake robot is constructed based on the structural model of the underwater snake robot, a state space and an action space are defined, and a physical constraint mapping function is designed; the motion control and decision model of the underwater snake robot includes a strategy network and a value network;

[0015] In a simulation environment, a target point and a target path are set, the underwater snake robot performs multi-waypoint path tracking, a composite reward function is designed, the motion control and decision model of the underwater snake robot is trained based on a reinforcement learning algorithm, and an optimal underwater snake robot cooperative control strategy is obtained;

[0016] Based on the optimal underwater snake robot cooperative control strategy, a multi-waypoint path tracking task of the underwater snake robot is performed, and cooperative control of the underwater snake robot is realized; the optimal underwater snake robot cooperative control strategy generates actual control quantities in real time according to the state of the underwater snake robot in the multi-waypoint path tracking task in combination with the physical constraint mapping function, and is used for control of the underwater snake robot.

[0017] Optionally, the central pattern generator phase update model is described by the following equation:

[0018] ;

[0019] wherein, denotes the time point, denotes the central pattern generator phase at the time point; denotes the time point, is the joint swing frequency of the underwater snake robot; is the circular constant; is the control time step;

[0020] Optionally, the joint periodic swing model is described by the following equation:

[0021] ;

[0022] wherein, is a sine calculation; denotes the central pattern generator phase at the time point, denotes the time point, is the joint number of the underwater snake robot, denotes the swing angle of the underwater snake robot at the time point, denotes the time point, denotes the swing angle of the underwater snake robot at the time point, The joint swing amplitude of the underwater snake robot is determined as The total number of joints of the underwater snake robot is determined as Pi is determined as The joint steering bias is determined as

[0023] Optionally, the tail single-propeller model describes the axial thrust of the tail propeller in the local coordinate system of the snake robot with respect to time , denoted as .

[0024] Optionally, the state space, the state vector of which includes: the x component of the linear velocity of the underwater snake robot in the world coordinate system, the yaw angle, the angular velocity z component, the sine value and the cosine value of the phase of the central pattern generator, the lateral offset distance of the underwater snake robot relative to the preset heading center line, and the heading error between the heading of the underwater snake robot and the target direction;

[0025] The action space, the action vector of which includes: the tail propeller thrust modulation amount, the joint swing amplitude modulation amount, the joint swing frequency modulation amount, and the joint steering bias modulation amount;

[0026] The physical constraint mapping function maps the action vector to the actual control amount.

[0027] Optionally, the target path is composed of a plurality of waypoints;

[0028] The composite reward function includes a propulsion progress reward, a centering reward, and a heading reward;

[0029] The propulsion progress reward is calculated based on the change amount of the distance between the underwater snake robot and the target point, and is mathematically expressed as follows:

[0030] ;

[0031] Wherein, represents the propulsion progress reward; , represents the distance between the underwater snake robot and the target point at the current time and the last time ; and represents the reward weight coefficient of the propulsion progress reward;

[0032] The centering reward encourages the underwater snake robot to maintain straight propulsion, and is mathematically expressed as follows:

[0033] ;

[0034] Wherein, represents the centering reward; represents the reward weight coefficient of the centering reward; represents an exponential function with natural constant as base; represents the lateral offset distance of the underwater snake-like robot to the current time target path centerline, which is calculated as follows:

[0035] ;

[0036] wherein, represents the position of the underwater snake-like robot at the current time ; represents the th waypoint, and together represent a pair of adjacent waypoints, and the path segment where the underwater snake-like robot is located at any time is described by the adjacent waypoints; represents the Euclidean norm, which is used to calculate the vector length; represents the projection coefficient on the path segment, which is defined as:

[0037] ;

[0038] wherein represents the square of the vector length; represents the truncation function, which is used to limit the input value in the interval ; the heading reward is mathematically represented as follows:

[0039] ;

[0040] wherein, represents the heading reward; represents the reward weight coefficient of the heading reward; represents the heading error between the heading of the underwater snake-like robot and the direction of the target point; represents the absolute value operation, which is used to represent the size of the heading error.

[0041] Optionally, the composite reward function further comprises:

[0042] time penalty term , which is a constant negative reward to avoid inefficient motion of the underwater snake-like robot;

[0043] waypoint reward , which is applied when the underwater snake-like robot reaches the waypoint;

[0044] success reward , which is applied when the underwater snake-like robot enters the preset distance range of the target point and terminates the current tracking round.

[0045] Optionally, the reinforcement learning algorithm is a policy gradient reinforcement learning algorithm for a continuous action space, and the policy network and the value network of a motion control and decision model of the underwater snake robot are optimized and trained by maximizing the expected cumulative return, and the trained policy network is used as an optimal underwater snake robot cooperative control strategy.

[0046] The application further provides an underwater snake robot single-propeller cooperative control system based on reinforcement learning, which is used for applying the underwater snake robot single-propeller cooperative control method based on reinforcement learning.

[0047] A robot model construction module is configured to establish a structure model of the underwater snake robot, wherein the underwater snake robot comprises a plurality of joint modules arranged in series along an axial direction and a tail single-propeller module; the joint module is internally provided with a central pattern generator, which is configured to cause periodic oscillation of each joint of the underwater snake robot, and to generate a propulsive force by interacting with a water body; and the tail single-propeller module is internally provided with a tail propeller, which is configured to provide an axial thrust.

[0048] A motion control and decision model construction module is configured to construct a motion control and decision model of the underwater snake robot based on the structure model of the underwater snake robot, to define a state space and an action space, and to design a physical constraint mapping function.

[0049] A simulation training module is configured to set a target point and a target path in a simulation environment, to cause the underwater snake robot to perform a multi-waypoint path tracking task, to design a reward function, to train the motion control and decision model of the underwater snake robot based on a reinforcement learning algorithm, and to obtain an optimal underwater snake robot cooperative control strategy and deploy the optimal underwater snake robot cooperative control strategy to a control execution module.

[0050] The control execution module is configured to execute the multi-waypoint path tracking task of the underwater snake robot based on the deployed optimal underwater snake robot cooperative control strategy, and to realize cooperative control of the underwater snake robot.

[0051] The application further provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to cause a processor to perform the cooperative control method.

[0052] By means of the above technical solution, the application provides an underwater snake robot single-propeller cooperative control method based on reinforcement learning, which has at least the following beneficial effects:

[0053] (1) The present application unifies the multi-joint wave propulsion of the snake robot provided by the central pattern generator and the axial thrust provided by the tail propeller into the reinforcement learning control framework, realizes the cooperative regulation between the two, avoids the problem of effective thrust loss caused by the change of the propeller thrust direction with the snake posture in the traditional control method, and improves the overall propulsion efficiency;

[0054] (2) The present application introduces CPG phase information (sine / cosine value) in the reinforcement learning state space, so that the control strategy can autonomously learn the "thrust-phase coordination" mechanism, and realize the application of high thrust only in the effective phase window of the tail alignment heading according to the characteristics of the underwater snake robot tail propeller follow-up swing, avoid the reverse yawing moment generated by the traditional fixed thrust strategy when the tail swings greatly, and significantly improve the underwater propulsion efficiency;

[0055] (3) The present application builds and trains the underwater snake robot motion control and decision model, unifies the multi-joint wave propulsion of the snake body from the CPG and the thrust output of the tail propeller into the control decision framework, without building an accurate hydrodynamic model, and through the reinforcement learning method, the complex nonlinear coupling relationship between the underwater snake robot motion and the underwater environment can be autonomously learned, the dependence of the control system on the modeling accuracy of the environment is reduced, and the adaptability in uncertain hydrodynamic environment is enhanced;

[0056] (4) In view of the characteristics of underwater non-supporting point and easy to drift sideways due to flow field interference, the present application guides the agent to learn to use "head bias" and "body posture adjustment" to generate hydrodynamic effect to offset environmental disturbance through a composite reward function, overcoming the limitations of traditional methods relying on accurate hydrodynamic modeling;

[0057] (5) The present application specially designs a physical constraint mapping mechanism conforming to underwater fluid dynamics in the action output link, which smoothes and limits the control instructions output by the strategy network according to the high damping and added mass characteristics of water medium, ensuring that the generated optimal strategy has feasibility and safety in the real underwater physical environment;

[0058] (6) The optimal underwater snake robot cooperative control strategy obtained by training can dynamically adjust the joint swing parameters and tail propelling thrust according to the real-time motion state of the underwater snake robot, can maintain motion stability in the underwater complex environment, and improves the control performance of the snake robot in the multi-waypoint track tracking task;

[0059] (7) The present application adopts a strategy optimization method in continuous action space, which is suitable for control scenes with multiple control quantity cooperative regulation, has good expansibility, and can be further applied to multi-propeller or higher degree of freedom underwater snake robot systems. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0061] Figure 1 A flow chart of a method for cooperative control of an underwater snake-like robot based on reinforcement learning;

[0062] Figure 2 A schematic diagram of a simulation test process for implementing a multi-waypoint path tracking task by an underwater snake-like robot in an embodiment of the application. DETAILED DESCRIPTION

[0063] In order to make the above objectives, characteristics and advantages of the application more apparent, comprehensible and easily understood, the application will be further described in detail below with reference to the drawings and specific embodiments. The implementation process of how the application applies technical means to solve technical problems and achieve technical effects can be fully understood and implemented by the application.

[0064] Those of ordinary skill in the art can understand that all or part of the steps of the embodiment method can be completed by programs instructing related hardware, and therefore, the application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can adopt a computer program product in the form of being implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0065] Please refer to Figures 1-2 , which shows a specific embodiment of the application. The embodiment establishes a structural model of an underwater snake-like robot, constructs a motion control and decision-making model of the underwater snake-like robot, defines a state space and an action space, unifies the multi-joint wave propulsion of the snake body and the thrust output of the tail propeller from the CPG into a control decision-making framework, performs path tracking of the underwater snake-like robot in a simulation environment, designs a composite reward function with path tracking error, heading deviation and propulsion efficiency as targets, trains the motion control and decision-making model of the underwater snake-like robot based on a reinforcement learning algorithm, generates an optimal underwater snake-like robot cooperative control strategy to perform a multi-waypoint path tracking task of the underwater snake-like robot, so that the cooperative control strategy of the joint wave form parameters and the propeller thrust is autonomously learned without relying on an accurate hydrodynamic model, thereby realizing stable and high-precision path tracking of the super-redundant underwater snake-like robot under the constraint of a multi-segment path containing a turning segment, and improving the motion stability and propulsion efficiency of the system.

[0066] The embodiment adopts Webots robot simulation platform as the motion simulation environment of underwater snake robot. The basic time step basicTimeStep is set to 32 ms in the simulation world file (.wbt), and the TIME_STEP=32 in the controller, so as to ensure that the simulation physical update frequency is consistent with the control period.

[0067] A flat ground node is set in the simulation environment to provide stable contact and motion support; no obstacles are additionally applied in the environment, but the motion trajectory to be tracked by the underwater snake robot is defined by a preset waypoint sequence (WAYPOINTS), i.e., a route point, to generate a target path. The underwater snake robot takes tracking the target path as a task target to perform motion control, so as to evaluate the stability and precision of the proposed control method in the multi-waypoint path tracking task.

[0068] The whole underwater snake robot is defined as a Robot node with the DEF name "underwater", and its structure includes the snake head, the joint modules of the snake body, and the tail single propeller module of the snake tail from front to back.

[0069] The snake body part is composed of 8 joint modules connected in series in the longitudinal direction, and the snake head and the first joint module and the adjacent joint modules are connected through rotary joints. Each rotary joint is configured with a rotary motor device, and the names are defined as Joint_1, Joint_2, …, Joint_8. The above joint numbering sequence is consistent with the physical sequence of the snake body from the head to the tail.

[0070] A tail single propeller module node is arranged at the tail of the underwater snake robot, and the tail single propeller module is defined with the DEF name "TAIL" to apply a continuously adjustable propelling force during simulation. The propelling force is applied by using the addForce interface provided by Webots, which is used to simulate the actual propelling effect of the underwater propeller under the condition of the posture change of the snake body.

[0071] After the controller is started, the handles of the underwater snake robot node and the device are obtained through the Webots Supervisor interface, including the underwater snake robot main node, the tail single propeller module node of the snake tail, and the joint motor devices.

[0072] In the initialization stage, all joint motors are set to speed control mode, and the initial angular velocity is set to zero to ensure that the joint swing angle is completely generated by the subsequent control logic.

[0073] Please refer to Figure 1 The embodiment proposes a single propeller cooperative control method for underwater snake robots based on reinforcement learning, which includes the following steps:

[0074] S1, establishing an underwater snake robot structure model.

[0075] The underwater snake robot structure model is used to describe the propulsion and turning process of the underwater snake robot in a water environment, the underwater snake robot comprising a plurality of joint modules arranged in series along an axis and a tail single propeller module, each joint module generating propulsion force by periodically oscillating and interacting with the water body, and the tail single propeller module being used to provide axial thrust while considering hydrodynamic resistance and joint movement constraints.

[0076] As a preferred embodiment of step S1, the specific process comprises:

[0077] S11, establishing a central pattern generator phase update model. The central pattern generator phase is updated over time, and the established central pattern generator phase update model is described by the following equation:

[0078] ;

[0079] wherein, denotes the time, denotes the central pattern generator phase at the time; is the joint oscillation frequency of the underwater snake robot; is the circular constant; is the control time step.

[0080] S12, establishing a joint periodic oscillation model. The oscillation angle of each joint of the underwater snake robot is generated by the central pattern generator, and the joint movement is described in the form of a sine wave based on the central pattern generator, and the joint periodic oscillation model is described by the following equation:

[0081] ;

[0082] wherein, is a sine calculation; is the joint number of the underwater snake robot, denotes the oscillation angle of the first joint of the underwater snake robot at the time; is the total number of joints of the underwater snake robot; is the joint turning bias, in this example, only acting on the rotation joint between the snake head and the first joint module behind it, for realizing the heading correction, and the corresponding of the remaining rotation joints are all set to 0.

[0083] ​​​S13, a tail single thruster model is established, on the basis of joint swing, an axial thrust along the local coordinate system of the robot is applied to the node of the tail single thruster module of the snake tail, so as to further enhance the propulsion efficiency and realize the cooperative control of joint swing and thruster output.

[0084] The tail single thruster model describes the axial thrust of the tail thruster under the local coordinate system of the snake robot with respect to time t, denoted as .

[0085] S2, based on the structure model of the underwater snake robot, a motion control and decision model of the underwater snake robot is constructed, the motion control and decision model of the underwater snake robot includes a strategy network and a value network. Meanwhile, the state space and the action space are defined, and a physical constraint mapping function is designed to perform physical action constraint mapping.

[0086] As a preferred embodiment of step S2, the definition of the state space and the action space, and the design of the physical constraint mapping function for physical action constraint mapping, specifically includes:

[0087] S21, in this embodiment, a reinforcement learning environment is constructed, and a state space is defined to comprehensively reflect the motion state of the underwater snake robot. Each state vector in the state space is defined as a 7-dimensional continuous vector, and the mathematical expression is as follows:

[0088] ;

[0089] Among them, , , respectively represent the linear velocity x component, the yaw angle, and the angular velocity z component of the underwater snake robot in the world coordinate system; represents the central pattern generator phase; , respectively represent the sine and cosine calculation; represents the lateral offset distance of the underwater snake robot relative to the preset heading center line, represents the heading error between the heading of the underwater snake robot and the target direction.

[0090] S22, the action space of the reinforcement learning is defined, wherein each action vector is defined as a 4-dimensional continuous vector, and the mathematical expression is as follows:

[0091] ;

[0092] Among them, represents the tail propulsion thrust modulation amount; represents the joint swing amplitude modulation amount; represents the joint swing frequency modulation amount; represents the joint steering bias modulation amount to realize the coordinated control of propulsion and steering behaviors in the path tracking process.

[0093] Through the above state information, the reinforcement learning agent can perceive the motion state, posture change and deviation from the target direction of the snake robot.

[0094] S23, design a physical constraint mapping function. The action output range of reinforcement learning is , which needs to be converted into actual control amount through linear mapping. The present application designs a physical constraint mapping function for each sub-component in the action vector as follows:

[0095] ;

[0096] wherein, , , , respectively represent the tail propulsion thrust modulation amount , the joint swing amplitude modulation amount , the joint swing frequency modulation amount , and the head joint steering bias modulation amount corresponding to the actual control amount; , , , respectively represent the tail propulsion thrust modulation amount , the joint swing amplitude modulation amount , the joint swing frequency modulation amount , and the head joint steering bias modulation amount corresponding to the physical constraint mapping function; represents the maximum thrust allowed by the tail propeller to output; represents the maximum value of the joint swing amplitude; represents the minimum value of the joint swing amplitude; represents the maximum value of the joint swing frequency; represents the minimum value of the joint swing frequency; represents the maximum steering bias angle allowed by the joint.

[0097] The application introduces CPG phase information (sine / cosine values) in the reinforcement learning state space, enables the control strategy to autonomously learn the "thrust-phase coordination" mechanism, and realizes the application of high thrust only in the effective phase window of the tail alignment heading of the underwater snake robot, thereby avoiding the reverse yawing moment generated by the traditional fixed thrust strategy when the tail swings greatly, significantly improving the underwater propulsion efficiency.

[0098] The application constructs and trains the underwater snake robot motion control and decision model, unifies the snake multi-joint wave propulsion from the CPG and the tail thruster thrust output into the control decision framework, does not need to construct an accurate hydrodynamic model, and can realize the autonomous learning of the complex nonlinear coupling relationship between the underwater snake robot motion and the underwater environment through the reinforcement learning method, thereby reducing the dependence of the control system on the environment modeling accuracy and enhancing the adaptability in the uncertain hydrodynamic environment.

[0099] S3, in the simulation environment, the target point and the target path are set, the underwater snake robot performs multi-waypoint path tracking, a composite reward function is designed, the underwater snake robot motion control and decision model is trained based on the reinforcement learning algorithm, and the optimal underwater snake robot cooperative control strategy is obtained.

[0100] In view of the characteristics of underwater non-supporting points and easy side slip drift caused by flow field interference, the application guides the agent to learn to use "head bias" and "body posture adjustment" to generate hydrodynamic effect to offset environmental disturbance through the composite reward function, and overcomes the limitation of the traditional method which depends on accurate hydrodynamic modeling.

[0101] The target path is composed of a plurality of waypoints, adjacent waypoints are defined 、 , and the path segment where the underwater snake robot is located at any time is described by the adjacent waypoints. The reward function is used to guide the reinforcement learning agent to learn the correct propulsion strategy, and the embodiment designs a diversified composite reward function, which comprehensively considers the efficiency, straight line keeping ability and time cost of the underwater snake robot advancing to the target point.

[0102] As a preferred embodiment of step S3, the specific process includes:

[0103] S31, a propulsion progress reward is designed based on the change of the distance of the underwater snake robot to the target point , and the mathematical expression is as follows:

[0104] ;

[0105] wherein, , denote the distance between the underwater snake robot and the target point at the current time , the last time , respectively. denotes the reward weight coefficient of the propulsion progress reward.

[0106] S32, design the centering reward , which is used to encourage the underwater snake robot to maintain straight propulsion, and the mathematical representation is as follows:

[0107] ;

[0108] wherein, denotes the reward weight coefficient of the centering reward; denotes the exponential function with the natural constant as the base; denotes the lateral offset distance of the underwater snake robot to the path centerline at the current time , and the calculation method is as follows:

[0109] ;

[0110] wherein, denotes the position of the underwater snake robot at the current time ; Euclidean norm, used to calculate the vector length; denotes the projection coefficient on the path segment, defined as:

[0111] ;

[0112] wherein denotes the square of the vector length; denotes the truncation function, used to limit the input value within the interval .

[0113] S33, design the heading reward , and the mathematical representation is as follows:

[0114] ;

[0115] wherein, denotes the reward weight coefficient of the heading reward; denotes the heading error between the underwater snake robot heading and the target point direction; denotes the absolute value operation, used to represent the size of the heading error.

[0116] S34, design time penalty term , constant negative reward for avoiding inefficient motion of the underwater snake robot.

[0117] S35, design path point reward , applied when the underwater snake robot reaches the way point.

[0118] S36, design success reward , applied when the underwater snake robot enters the target point within the preset distance range and terminates the current tracking round.

[0119] S37, push progress reward , centering reward , heading reward , time penalty term , path point reward , success reward , the total reward at time is composed of , mathematically represented as follows:

[0120] .

[0121] S38, policy gradient reinforcement learning algorithm based on continuous action space, which optimizes and trains the policy network and value network of the underwater snake robot motion control and decision-making model by maximizing the expected cumulative return. In this embodiment, the Proximal Policy Optimization (PPO) algorithm in the Stable Baselines3 framework is used to train the underwater snake robot motion control and decision-making model.

[0122] Both the policy network (Actor) and the value network (Critic) are constructed using a multi-layer perceptron (MLP) structure. The policy network is used to output a continuous action vector based on the current state space, and the value network is used to evaluate the value of the current state space.

[0123] The PPO algorithm clips the policy update amplitude to ensure the stability of the policy network update, and the loss function of the clipped policy network is as follows:

[0124] ;

[0125] where, represents the minimum value operation; represents the expected operation on the data at time ; represents clipping the policy ratio to interval, to limit the update range; a clipping threshold representing policy update, to limit the policy update range; representing the current policy network parameters; policy ratio is defined as:

[0126] ;

[0127] wherein, is the action vector and the state vector of the policy network; representing the policy network parameters before update.

[0128] advantage function is calculated using the generalized advantage estimation method:

[0129] ;

[0130] wherein, representing the value network, is the value network parameter.

[0131] The total loss function of the underwater snake robot motion control and decision model training is represented as:

[0132] ;

[0133] wherein, is the value network loss function, is the coefficient of this loss term; representing the policy under the state state vector , to avoid the policy network converging to a suboptimal deterministic action too early, is the coefficient of this loss term.

[0134] The PPO algorithm is first initialized at the beginning of training, and the underwater snake robot interacts with the simulation environment in the form of an agent during the training process, collects state, action and reward data, calculates the reward function, and iteratively updates the policy network and value network parameters through the PPO algorithm. When the network parameters converge or reach the preset maximum tracking round, the training is ended.

[0135] After training is completed, the policy network obtained by training is loaded, and the simulation test of the underwater snake robot for the multi-waypoint path tracking task is performed, and the process can be referred to Figure 2 , Figure 2The continuous time sequence snapshots of the motion process of the underwater snake robot from top to bottom are displayed, and the preset path point positions are marked by green dots, and the path tracking direction is a black arrow, the motion state of the underwater snake robot along the preset path point sequence is observed in a real-time visualization manner, and the stability and accuracy of the underwater snake robot in the multi-path point path tracking task are verified.

[0136] The optimal underwater snake robot cooperative control strategy obtained by training can dynamically adjust the joint swing parameters and the tail propulsion thrust according to the real-time motion state of the underwater snake robot, can maintain the motion stability in the underwater complex environment, and improves the control performance of the snake robot in the multi-path point path tracking task.

[0137] The application adopts the strategy optimization method of continuous action space, is suitable for the control scene of multi-control quantity cooperative adjustment, has good expansibility, and can be further applied to the underwater snake robot system with multiple propellers or higher degrees of freedom.

[0138] S4, based on the optimal underwater snake robot cooperative control strategy, an underwater snake robot multi-path point path tracking task is performed.

[0139] The optimal underwater snake robot cooperative control strategy is deployed, the actual control quantity is generated in real time according to the state of the underwater snake robot in the multi-path point path tracking task, and the cooperative control of the underwater snake robot is realized.

[0140] The application specially designs a physical constraint mapping mechanism conforming to underwater fluid dynamics in the action output link, the control instruction output by the strategy network is smoothed and limited in amplitude for the high damping and additional mass characteristics of the water medium, so that the generated optimal strategy has feasibility and safety in the real underwater physical environment.

[0141] The embodiment of the application also provides an underwater snake robot single-propeller cooperative control system based on reinforcement learning, which is used for applying the cooperative control method, and comprises:

[0142] A robot model construction module is used to establish a structure model of the underwater snake robot, wherein the underwater snake robot comprises a plurality of joint modules arranged in series along an axial direction and a tail single-propeller module; the joint module is internally provided with a central pattern generator, which is used to make the joints of the underwater snake robot swing periodically and interact with water to generate propulsion; and the tail single-propeller module is internally provided with a tail propeller, which is used to provide axial thrust.

[0143] The motion control and decision model construction module is configured to construct a motion control and decision model of the underwater snake robot based on a structure model of the underwater snake robot, define a state space and an action space, and design a physical constraint mapping function.

[0144] The simulation training module is configured to set a target point and a target path in a simulation environment, make the underwater snake robot perform a multi-waypoint path tracking task, design a reward function, train the motion control and decision model of the underwater snake robot based on a reinforcement learning algorithm, and obtain an optimal underwater snake robot cooperative control strategy and deploy the optimal underwater snake robot cooperative control strategy to the control execution module.

[0145] The control execution module is configured to execute the multi-waypoint path tracking task of the underwater snake robot based on the deployed optimal underwater snake robot cooperative control strategy, and realize cooperative control of the underwater snake robot.

[0146] The application also provides a computer readable storage medium storing computer instructions for causing a processor to execute the base cooperative control method.

[0147] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0148] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor, or other system that can take instructions from an instruction execution system, apparatus or device, or in conjunction with these instructions execution systems, apparatus or devices.

[0149] The above embodiments have been described in detail, and the principles and embodiments of the present application have been described herein by applying specific examples; the above embodiment descriptions are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific embodiments and application scope will be changed; in view of the above, the content of the present application should not be understood as a limitation.

Claims

1. A single-thruster cooperative control method for an underwater snake-like robot based on reinforcement learning, characterized in that, include: Establish a structural model of the underwater snake-like robot; The underwater snake-like robot structural model includes: A phase update model for the central pattern generator describes the phase of the central pattern generator. A joint periodic oscillation model was developed to calculate the oscillation angles of each joint in an underwater snake robot. A single tail thruster model describes the axial thrust provided by the tail thruster; An underwater snake robot motion control and decision-making model is constructed based on the underwater snake robot structural model, defining the state space and action space, and designing physical constraint mapping functions; the underwater snake robot motion control and decision-making model includes a policy network and a value network; In the simulation environment, target points and target paths are set, and underwater snake robots perform multi-waypoint path tracking. A composite reward function is designed, and the underwater snake robot motion control and decision-making model is trained based on reinforcement learning algorithm to obtain the optimal underwater snake robot cooperative control strategy. The underwater snake robot performs multi-waypoint path tracking tasks based on the optimal underwater snake robot cooperative control strategy, thereby achieving cooperative control of the underwater snake robot. The optimal underwater snake robot cooperative control strategy generates actual control quantities in real time based on the state of the underwater snake robot in the multi-waypoint path tracking task, combined with the physical constraint mapping function, for the control of the underwater snake robot.

2. The collaborative control method according to claim 1, characterized in that: The phase update model of the central pattern generator is described by the following equation: ； in, Indicates time, express Phase of the central pattern generator at any given moment; The joint oscillation frequency of the underwater snake robot; Pi; To control the time step.

3. The collaborative control method according to claim 1, characterized in that: The periodic oscillation model of the joint is described by the following equation: ； in, Calculate the sine; express Phase of the central pattern generator at any given moment; Number the joints of the underwater snake robot. The underwater snake-like robot is the first Each joint The angle of swing at any given moment; This refers to the amplitude of joint oscillation in an underwater snake-like robot. This represents the total number of joints in the underwater snake-like robot. Pi; This is for joint steering offset.

4. The cooperative control method according to claim 1, characterized in that: The tail single thruster model describes the tail thruster's position relative to time in the snake robot's local coordinate system. The axial thrust, denoted as .

5. The collaborative control method according to claim 1, characterized in that: The state space, whose state vector includes: the x-component of the linear velocity of the underwater snake robot in the world coordinate system, the yaw angle, the z-component of the angular velocity, the sine and cosine values ​​of the phase of the central mode generator, the lateral offset distance of the underwater snake robot relative to the preset heading centerline, and the heading error between the heading of the underwater snake robot and the target direction. The motion space, whose motion vectors include: tail thrust modulation amount, joint swing amplitude modulation amount, joint swing frequency modulation amount, and joint steering offset modulation amount; The physical constraint mapping function maps the action vector to the actual control quantity.

6. The collaborative control method according to claim 1, characterized in that: The target path consists of several waypoints; The composite reward function includes progress reward, centering reward, and orientation reward; The progress reward is calculated based on the change in distance between the underwater snake-like robot and the target point, and is mathematically represented as follows: ； in, Indicates a reward for progress; , These represent the underwater snake robot at the current moment. The previous moment Distance to the target point; This indicates the reward weighting coefficient for progress rewards; The centering reward encourages the underwater snake-like robot to maintain straight-line propulsion, mathematically represented as follows: ； in, This indicates a middle-ground reward; This represents the reward weighting coefficient for the center reward; Represented by natural constant An exponential function with base 0; This indicates the underwater snake robot at the current moment. The lateral offset distance of the target path centerline is calculated as follows: ； in, This indicates the underwater snake-like robot at the current moment. Location; Indicates the first One waypoint, and A pair of adjacent waypoints are collectively represented, and the path segment where the underwater snake robot is located at any given time is described by the adjacent waypoints; This represents the Euclidean norm, used to calculate the magnitude of a vector. The projection coefficients on the path segment are defined as follows: ； in Represents the square of the vector's magnitude; This represents the truncation function. Used to restrict input values ​​to a range The orientation reward mathematical representation is as follows: ； in, Indicates a direction towards reward; This represents the reward weighting coefficient that influences the outcome of a reward program. This indicates the heading error between the heading of the underwater snake robot and the direction of the target point. This represents absolute value operations and is used to characterize the magnitude of heading error.

7. The cooperative control method according to claim 6, characterized in that: The composite reward function also includes: Time penalty items This provides a constant negative reward, thus avoiding the inefficient movement of underwater snake robots. Path point rewards Apply when the underwater snake robot reaches a waypoint; Success reward When the underwater snake robot enters the target point within a preset distance range, the current tracking round is applied and terminated.

8. The collaborative control method according to claim 1, characterized in that: The reinforcement learning algorithm is a policy gradient reinforcement learning algorithm in continuous action space. It optimizes the policy network and value network of the underwater snake robot motion control and decision-making model by maximizing the expected cumulative reward. The trained policy network serves as the optimal cooperative control strategy for the underwater snake robot.

9. A single-thruster cooperative control system for an underwater snake robot based on reinforcement learning, used to apply the cooperative control method as described in any one of claims 1-8, characterized in that, include: A robot model building module is used to establish a structural model of an underwater snake-like robot. The underwater snake-like robot includes multiple joint modules arranged in series along the axial direction and a single tail thruster module. Each joint module has a built-in central pattern generator, which causes the joints of the underwater snake-like robot to oscillate periodically, interacting with the water to generate propulsion. The single tail thruster module has a built-in tail thruster, which provides axial thrust. The motion control and decision-making model construction module is used to construct the motion control and decision-making model of the underwater snake robot based on the underwater snake robot's structural model, define the state space and action space, and design the physical constraint mapping function; The simulation training module is used to set target points and target paths in a simulation environment, and to perform multi-waypoint path tracking tasks for the underwater snake robot. It designs a reward function, trains the motion control and decision-making model of the underwater snake robot based on reinforcement learning algorithm, obtains the optimal cooperative control strategy of the underwater snake robot, and deploys it to the control execution module. The control execution module is used to execute the underwater snake robot's multi-waypoint path tracking task based on the deployed optimal underwater snake robot cooperative control strategy, thereby realizing the cooperative control of the underwater snake robot.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the cooperative control method according to any one of claims 1-8.

Images