Underwater hexapod robot multi-wall continuous motion control method based on reinforcement learning

Abstract

The invention provides an underwater hexapod robot multi-wall surface continuous motion control method based on reinforcement learning, and the method comprises the steps: obtaining a forward distance from the mass center of an underwater hexapod robot to a to-be-climbed wall surface through distance measurement equipment, and obtaining a distance from the mass center of the underwater hexapod robot to the current wall surface of the underwater hexapod robot through the distance measurement equipment; the method comprises the following steps: setting a state space, an action space and a reward function for stable crossing and constant-speed directional climbing tasks by analyzing the constant-speed directional climbing and wall surface crossing tasks; a deep deterministic strategy gradient reinforcement learning method is utilized to train a gait of stably crossing between wall surfaces with different dip angles. According to the method, the problems that a traditional gait planning method is complex in modeling and unstable in gait switching are solved, the underwater hexapod robot can cross over wall faces with different dip angles and conduct directional and constant-speed climbing along the wall faces with the different dip angles, meanwhile, the thruster thrust adaptive to the torque of the blade point leg can be solved, and the aim that the robot can climb on the wall faces with the different dip angles is achieved. And the motion stability of the underwater hexapod robot is ensured.

Description

Technical Field

[0001] The invention relates to the technical field of underwater hexapod robots, and in particular to a wall climbing and crossing gait planning and control method for an underwater hexapod robot. Background Art

[0002] In recent years, autonomous underwater vehicles (AUVs) and unmanned remotely operated vehicles (ROVs) have been widely used in ocean exploration, development and utilization. However, traditional AUVs and ROVs are only driven by thrusters, and they cannot interact with the environment, which means they are not suitable for performing localized fine-scale operations. In order to improve the operational capability of marine equipment, hybrid drive of thrusters and legs has become a solution.

[0003] Accurate mathematical models can fully describe the motion of legged robots, such as the spring-loaded inverted pendulum model. However, model-based gait planning methods are computationally complex and generally have unmodeled dynamics. Gait planning methods based on bionics can effectively circumvent the above problems. With the continuous development of central pattern generator (CPG) theory, many different oscillator models and CPG networks have been applied to gait planning. However, existing CPG-based methods mostly produce periodic gaits, which greatly limits the maneuverability of legged robots. How to design non-periodic mappings to generate more flexible and autonomous gaits is a very challenging topic. The core of reinforcement learning is goal-oriented learning from interaction, which is used to solve sequential decision-making problems. Continuous motion on different walls can be classified as a sequential decision-making problem, that is, how to coordinate the control instructions of the legs and thrusters so that the robot can smoothly transition and climb between walls of different inclinations while maintaining the required speed and yaw angle. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a multi-wall continuous motion control method for an underwater hexapod robot based on reinforcement learning. The present invention designs an adaptive gait planning and control method based on reinforcement learning. The present invention is not based on a model, avoids complex calculations and gait switching, and autonomously generates and adjusts the gait during the robot's motion. At the same time, through certain force analysis, the thrust of the propeller adapted to the blade leg joint torque is solved, so that the underwater hexapod robot can smoothly switch between walls with different inclination angles and stably climb along the inclined wall.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] The present invention provides a reinforcement learning-based underwater hexapod robot wall climbing and crossing gait planning and control method. The underwater hexapod robot comprises 8 propellers, 6 blade legs and a robot body, wherein 4 propellers are vertically distributed, 4 propellers are horizontally vector-distributed, and the 8 propellers are all embedded in the robot body; the 6 blade legs are semicircular blade legs, and the blade leg hip joints are connected to 6 driving motors on the left and right sides of the robot body, comprising the following steps:

[0007] S1: For a wall with a known inclination angle, use a distance measuring device to obtain the forward distance from the center of mass of the underwater hexapod robot to the wall to be climbed, and at the same time use a distance measuring device to obtain the distance between the center of mass of the underwater hexapod robot and the wall where the underwater hexapod robot is currently located;

[0008] S2: By analyzing the constant speed directional climbing and wall crossing tasks, the state space, action space and reward function for the smooth crossing and constant speed directional climbing tasks are set; the directional constant speed means that the robot can move at the expected forward speed and the expected heading angle, and the smooth crossing means that the robot can achieve continuous movement between the sudden inclination walls while maintaining stability;

[0009] S3: Using the deep deterministic policy gradient (DDPG) reinforcement learning method, we trained an underwater hexapod robot to climb along walls of different inclinations at a constant speed and in a directional manner, and to smoothly cross between walls of different inclinations.

[0010] S4: In order to improve the stability of the underwater hexapod robot's movement, on the one hand, a basic thrust is applied to the underwater hexapod robot's propeller to ensure that the blade legs of the underwater hexapod robot do not slip during movement. On the other hand, in order to reduce the influence of the restoring force and restoring torque on the movement of the underwater hexapod robot, additional thrust is applied to the underwater hexapod robot's propeller according to the underwater hexapod robot's current attitude angle to overcome the restoring force and restoring torque.

[0011] In S1, for a wall with a known inclination angle, a distance measuring device fixed to the mass center of the underwater hexapod robot is used to obtain the forward distance d from the mass center of the underwater hexapod robot to the wall to be climbed, and a Doppler velocimeter fixed to the mass center of the underwater hexapod robot is used to obtain the height h of the mass center of the underwater hexapod robot relative to the wall where the robot is currently located. Whether the blade leg touches the ground is determined by the height h and the joint angle q of each blade leg of the underwater hexapod robot. i The judgment is as follows:

[0012]

[0013] Among them, a i represents the ground contact state of the i-th blade leg, a i =1 indicates that the i-th blade leg touches the ground, a i=0 indicates that the i-th blade leg has not touched the ground, i is the number of the blade leg, i=1,2,…,6; l r is the blade leg radius; θ are the inclination angle of the wall where the i-th blade leg of the underwater robot is located and the pitch angle of the underwater hexapod robot; b x pi and b y pi They respectively represent the coordinate components of the landing point of the underwater hexapod robot along the x and y directions in the body coordinate system.

[0014] In step S2, by analyzing the constant speed directional climbing and wall crossing tasks, the state space, action space and reward functions for the smooth crossing and constant speed directional climbing tasks are set; the specific steps are as follows:

[0015] Considering the task requirements and robot characteristics, a 15-dimensional column vector is selected as the state space S, that is, Where T is the matrix transpose symbol; v x are the expected forward velocity and actual forward velocity of the underwater hexapod robot, respectively; ψ d , ψ are the expected heading angle and the actual heading angle of the underwater hexapod robot, respectively; θ are the inclination angle of the wall to be climbed and the pitch angle of the underwater hexapod robot; is the rolling angle of the underwater hexapod robot; the action set is defined as in is the joint angular velocity of the i-th blade leg;

[0016] In order to enable the underwater hexapod robot to achieve constant speed and directional climbing, and to smoothly cross between walls with different inclination angles, the reward function is set as:

[0017]

[0018] Among them, λ1,λ2,…,λ8 are parameters to be adjusted that are greater than zero, t represents the current time, s t is the state quantity at the current moment, a t is the action executed at the current moment, g(t) is the characterization function of the underwater hexapod robot's ground contact stability, c1(t) is the switch signal of whether to start crossing the wall, c2(t) is the switch signal of whether to climb the wall, and u(t) is the characterization function of the blade leg utilization rate; the first four items of the reward function are set according to the error of the state variables in the state space to achieve constant speed orientation and posture stability; the last four items are functions of the state quantity and action quantity, which are used to guide the robot to learn to cross stably, and their specific definitions are as follows:

[0019] In order to make the support area formed by the support legs greater than zero, the number of support legs of the underwater hexapod robot needs to be greater than or equal to 3, and there must be support legs on both sides of the underwater hexapod robot body. Therefore, the ground contact stability characterization function g is defined as:

[0020]

[0021] in,

[0022] In order to achieve the smooth crossing of the underwater hexapod robot between walls with different inclination angles, on the one hand, the underwater hexapod robot should be able to approach the wall to be climbed, and on the other hand, the underwater hexapod robot should be able to climb up the wall of another inclination angle. Therefore, the switch signals c1(t) and c2(t) are set in the reward function, as follows:

[0023]

[0024]

[0025] Among them, D is the distance threshold, Θ is the pitch angle threshold, and the value is adjusted according to the training effect.

[0026] In addition, in order to drive the six blade legs evenly and avoid the situation where individual legs are stationary for a long time, it is necessary to consider the extreme difference constraint of the blade leg joint angular velocity at each moment to improve the utilization rate of the blade legs. The blade leg utilization characterization function u(t) is defined as:

[0027]

[0028] in They represent the maximum and minimum angular velocities of the six blade leg joints at time t respectively. Q represents the extreme difference threshold of angular velocity, and its value is adjusted according to the training situation.

[0029] In the step S3, a reinforcement learning method of Deep Deterministic Policy Gradient (DDPG) is used to train a gait that enables the underwater hexapod robot to climb along walls of different inclinations at a constant speed and in a directional manner and to smoothly cross between walls of different inclinations;

[0030] The DDPG algorithm is based on the actor-critic framework and is a different-strategy reinforcement learning method, including an actor neural network and a critic neural network. The actor neural network and the critic neural network each contain an evaluation network and a target network. The so-called different-strategy means that the evaluation strategy actually executed during the training process is not the same as the current optimal target strategy, so as to avoid the network parameters being used to calculate gradients while being frequently updated, so as to improve the stability of the training process and make the training easy to converge; the target strategy and its corresponding optimal value function are generated by the target network, and the target network parameters are obtained by soft updating of the evaluation network parameters. Therefore, the target network parameters change less, so the target network is used to calculate the gradient, and the evaluation strategy is generated by the evaluation network, and noise is added to convert it into a random strategy, which interacts with the environment and updates the evaluation network parameters through the gradient calculated by the target network. New experience is accumulated in each interaction, so as to ensure sufficient exploration.

[0031] In the step S4, in order to improve the stability of the underwater hexapod robot's movement, on the one hand, a basic thrust is applied to the underwater hexapod robot's propeller to ensure that the blade legs of the underwater hexapod robot do not slip during movement; on the other hand, in order to reduce the influence of the restoring force and the restoring torque on the movement of the underwater hexapod robot, an additional thrust is applied to the underwater hexapod robot's propeller according to the current attitude angle of the underwater hexapod robot to overcome the restoring force and the restoring torque;

[0032] According to the blade leg joint torque τ of the generated gait i , we can get:

[0033]

[0034] Among them, F τix 、F τiy are the components of the force acting on the touchdown point by the blade leg joint torque along the x and y directions in the touchdown point coordinate system, respectively. r is the blade leg radius;

[0035] According to the no-slip constraint, the vertical force F required for the blade leg to touch the point is calculated. Tiy , the specific calculation formula is as follows:

[0036]

[0037] Where μ is the ground friction coefficient; N i is the ground support force at the touchdown point;

[0038] According to the DH coordinate transformation, the vertical force F required for the blade leg to touch the ground is Tiy The force required to convert to the center of mass of the robot is provided by the thruster. The thrust components of the thruster in the three directions of the body coordinate system are recorded as T x , Ty 、T z , then F Tiy and T x 、T y 、T z Satisfies the following relationship:

[0039]

[0040] Therefore, the thruster base thrust for:

[0041]

[0042] in, is the basic thrust value of the i-th thruster; C t is the thrust allocation matrix;

[0043] The restoring force and restoring torque of the underwater hexapod robot are determined by gravity G, buoyancy B and attitude angle. The components of the restoring force in the x, y and z directions are denoted as F gx 、F gy 、F gz The components of the restoring moment in the x, y, and z directions are denoted as M gx 、M gy 、M gz , then the six components are specifically expressed as:

[0044]

[0045] Among them, x c ,y c ,z c is the coordinate of the buoyancy center of the underwater robot in the body coordinate system;

[0046] Therefore, the additional thrust of the propeller for:

[0047]

[0048] in, is the additional thrust value of the i-th thruster; C t is the thrust allocation matrix;

[0049] In summary, the total thrust value of the i-th thruster is

[0050] The beneficial effects of the present invention are mainly reflected in the following two aspects:

[0051] (1) Aiming at the multi-task operation scenario of the underwater hexapod robot, a reward function is designed so that the trained gait can take into account multiple tasks such as orientation and speed control, wall climbing, and wall crossing, thus avoiding the problems of complex modeling and unstable gait switching in traditional gait planning methods.

[0052] (2) For the hybrid drive underwater hexapod robot, a reinforcement learning-based gait planning method for the underwater hexapod robot is proposed, which enables the underwater hexapod robot to achieve the tasks of crossing between walls with different inclination angles and climbing along walls with different inclination angles in a directional and constant speed manner. At the same time, the thrust of the propeller adapted to the blade leg torque can be solved to ensure the motion stability of the underwater hexapod robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the wall climbing and crossing gait planning and control method of the underwater hexapod robot based on reinforcement learning.

[0054] Figure 2 (a) is a schematic diagram of the underwater hexapod robot in the Gazebo simulation platform. Figure 2 (b) is a schematic diagram of the inclined wall under the Gazebo simulation platform.

[0055] Figure 3 Schematic diagram of the location of the hip joints of the six legs of the underwater hexapod robot.

[0056] Figure 4 Schematic diagram of the positions of the six thrusters of the underwater hexapod robot.

[0057] Figure 5 Schematic diagram of the neural network training process in deep reinforcement learning.

[0058] Figure 6 It is the reward function change curve in the reinforcement learning algorithm.

[0059] Figure 7 This is the three-dimensional motion trajectory of the underwater hexapod robot when climbing.

[0060] Figure 8 (a) and (b) are the horizontal and vertical motion trajectories of the underwater hexapod robot during climbing, respectively.

[0061] Fig. 9 (a) and (b) are the heading angle change curve and heading angle error change curve of the underwater hexapod robot, respectively.

[0062] Fig.10 (a) and (b) are the pitch angle change curve and pitch angle error change curve of the underwater hexapod robot, respectively.

[0063] Fig.11(a) and (b) are the forward velocity change curve and forward velocity error change curve of the underwater hexapod robot, respectively.

[0064] Fig.12 This is the thrust variation curve of the underwater hexapod robot propeller.

[0065] Fig.13 (a) to Fig.13 (h) Schematic diagram of the movement process of the underwater hexapod robot in the Gazebo simulation platform. DETAILED DESCRIPTION

[0066] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0067] The present invention proposes a gait planning and control method based on reinforcement learning for an underwater hexapod robot driven by a hybrid of a thruster and a semicircular blade leg. The method adaptively adjusts the joint angles of the blade legs and the thrust of the thruster to enable the underwater hexapod robot to perform the task of crossing between walls with different inclination angles, and to climb walls with different inclination angles at a constant speed and in a directional manner. The method includes the following steps: S1: For a wall with a known inclination angle, a distance measuring device is used to obtain the forward distance from the center of mass of the underwater hexapod robot to the wall to be climbed, and at the same time, a distance measuring device is used to obtain the distance between the center of mass of the underwater hexapod robot and the wall where the underwater hexapod robot is currently located; S2: By analyzing the constant speed and directional climbing and wall crossing tasks, the state space, action space and reward functions for the tasks of smooth crossing and constant speed and directional climbing are set; S3: Using the deep deterministic policy gradient (DDPG) reinforcement learning method, a method that can enable The underwater hexapod robot can achieve constant speed and directional climbing along walls with different inclination angles and can smoothly cross between walls with different inclination angles in a gait; S4: In order to improve the stability of the underwater hexapod robot's movement, on the one hand, a basic thrust is applied to the underwater hexapod robot's propeller to ensure that the underwater hexapod robot's blade legs do not slip during movement; on the other hand, in order to reduce the influence of the restoring force and restoring torque on the movement of the underwater hexapod robot, according to the current attitude angle of the underwater hexapod robot, an additional thrust is applied to the underwater hexapod robot's propeller to overcome the restoring force and restoring torque. The gait planning and control method proposed in the present invention can enable the robot to cross and climb walls with different inclination angles by adaptively adjusting the gait and propeller thrust.

[0068] Therefore, the present invention will design a gait planning and control method based on reinforcement learning. This method is not based on a model, avoids complex calculations and gait switching, and at the same time, through certain force analysis, solves the thrust of the propeller that is adapted to the torque of the blade leg joint, so that the underwater hexapod robot can smoothly switch between different inclination angles and stably climb along the inclined wall.

[0069] In view of the problem that the existing gait planning methods have complex modeling and poor adaptability, the DDPG reinforcement learning algorithm of the present invention proposes an adaptive gait planning and control method for wall climbing and crossing tasks. The following difficulties exist in designing gait planning and control methods for wall climbing and crossing: 1. For multiple tasks such as constant speed directional climbing and wall crossing, how to set the reward function so that it can accurately reflect the task requirements; 2. For continuous control problems with higher-dimensional action spaces, how to improve training efficiency; 3. How to improve the adaptability of gait planning so that the underwater hexapod robot can smoothly cross and climb different walls.

[0070] The present invention will now be further described with reference to the embodiments and the accompanying drawings:

[0071] Figure 1 The following is a schematic diagram of a reinforcement learning-based underwater hexapod robot wall climbing and crossing gait planning and control method. First, the expected climbing speed v of the underwater hexapod robot is set. xd and the desired heading angle ψ d , and using the current climbing speed v of the underwater hexapod robot x and heading angle ψ, obtain the heading angle error and climbing speed error, and at the same time, based on the ground contact status of each blade leg, construct a representation function of the underwater hexapod robot's stable climbing. Secondly, construct a reward function r that includes heading speed error, attitude angle constraint and stability representation t , where the heading speed error ensures that the underwater hexapod robot walks at the desired heading speed; the attitude angle constraint and the distance constraint from the wall ensure that the underwater hexapod robot can approach and climb the wall; the stability representation ensures that the underwater hexapod robot touches the ground and walks stably; the blade leg joint angular velocity extreme difference constraint ensures that each blade leg is driven evenly, thereby avoiding the situation where individual legs are stationary for a long time. Finally, based on the DDPG reinforcement learning method, the target neural network is trained to obtain the underwater hexapod robot to achieve the desired heading ψ d and speed v xd The cross-angle is When climbing steadily on the wall, the state variable climbing speed v x , heading angle ψ, pitch angle θ, six-leg rotation angle q1,…,q6, rangefinder measurement value d and six-leg angular velocity The mapping relationship between them.

[0072] Figure 2 (a) and (b) are schematic diagrams of the underwater hexapod robot and the inclined wall in the Gazebo simulation platform. In the Gazebo simulation platform, the models of the underwater hexapod robot and the inclined wall are constructed for simulation. Figure 3 Schematic diagram of the locations of the hip joints of the six legs of the hybrid-drive underwater hexapod robot. Figure 4Schematic diagram of the positions of the six thrusters of the hybrid-drive underwater hexapod robot.

[0073] Since the kinematic model of the underwater hexapod robot is relatively complex, and the CPG-based gait planning method can only generate periodic or approximately periodic gaits, the existing model-based optimization or random search methods and CPG-based gait planning methods are difficult to use for operations involving crossing between walls with different inclination angles. To this end, the present invention proposes to use a deep reinforcement learning algorithm to plan the crossing and climbing gaits of the hexapod robot, which specifically includes the following steps:

[0074] S1: For a wall with a known inclination angle, use a distance measuring device to obtain the forward distance from the center of mass of the underwater hexapod robot to the wall to be climbed, and use the distance measuring device to obtain the distance between the center of mass of the underwater hexapod robot and the wall where the underwater hexapod robot is currently located;

[0075] S2: By analyzing the constant speed directional climbing and wall crossing tasks, the state space, action space and reward function for smooth crossing and constant speed directional climbing tasks are set;

[0076] S3: Using the DDPG-based reinforcement learning method, a gait is trained that enables the underwater hexapod robot to climb along walls of different inclinations at a constant speed and in a directional manner, and to smoothly cross between walls of different inclinations.

[0077] S4: In order to improve the stability of the underwater hexapod robot's movement, on the one hand, a basic thrust is applied to the underwater hexapod robot's propeller to ensure that the blade legs of the underwater hexapod robot do not slip during movement. On the other hand, in order to reduce the influence of the restoring force and restoring torque on the movement of the underwater hexapod robot, additional thrust is applied to the underwater hexapod robot's propeller according to the underwater hexapod robot's current attitude angle to overcome the restoring force and restoring torque.

[0078] To solve the problem of an underwater hexapod robot crossing walls with different inclination angles, the present invention uses a rangefinder installed at the center of mass to measure the forward distance between the center of mass of the underwater hexapod robot and the inclined wall. At the same time, a Doppler velocimeter fixed to the center of mass of the underwater hexapod robot is used to obtain the height of the center of mass of the underwater hexapod robot relative to the wall where the robot is currently located. A threshold is set to determine whether to start crossing the wall, and the signal is introduced into the reward function. At the same time, the a priori known wall inclination information is used, and the DDPG reinforcement learning algorithm is adopted to learn the mapping relationship between the underwater hexapod robot's smooth crossing of walls with different inclination angles, the directional and constant speed climbing of walls and the wall inclination, and the distance between the robot's center of mass and the wall.

[0079] In order to make the support area formed by the support legs greater than zero, the number of support legs of the underwater hexapod robot must be greater than or equal to 3, and there must be support legs on both sides of the underwater hexapod robot body. The expression for whether a stable support area can be formed is:

[0080]

[0081] in, a i represents the ground contact state of the i-th blade leg, a i =1 indicates that the i-th blade leg touches the ground, a i =0 indicates that the i-th blade leg has not touched the ground, i is the number of the blade leg, i=1, 2, ..., 6.

[0082] Considering the task requirements and robot characteristics, a 15-dimensional column vector is selected as the state space S, that is, Where T is the matrix transpose symbol; q i is the joint angle of the i-th blade leg of the underwater hexapod robot; d is the distance measurement value of the distance measuring device at the center of mass of the underwater hexapod robot, that is, the distance between the underwater robot and the wall to be climbed; h is the height of the center of mass of the underwater hexapod robot relative to the wall where the underwater hexapod robot is currently located; v x are the expected forward velocity and actual forward velocity of the underwater hexapod robot, respectively; ψ d , ψ are the expected heading angle and the actual heading angle of the underwater hexapod robot, respectively; θ are the inclination angle of the wall to be climbed and the pitch angle of the underwater hexapod robot; is the rolling angle of the underwater hexapod robot. The action set is defined as in is the joint angular velocity of the i-th blade leg.

[0083] In order to enable the underwater hexapod robot to achieve constant speed and directional climbing, and to smoothly cross between walls with different inclinations, the reward function is set as

[0084]

[0085] Among them, λ1…λ8 are coefficients greater than zero, t represents the current time, s t is the state quantity at the current moment, a t is the action executed at the current moment, g(t) is the characterization function of the underwater hexapod robot's ground contact stability, c1(t) is the switch signal of whether to start crossing the wall, c2(t) is the switch signal of whether to climb the wall, and u(t) is the characterization function of the blade leg utilization rate. The first four items of the reward function are set according to the error of the state variables in the state space to achieve constant speed orientation and posture stability. The last four items are functions of the state quantity and action quantity, which are used to guide the robot to learn to cross stably. Their specific definitions are as follows.

[0086] In order to achieve the smooth crossing of the underwater hexapod robot between walls with different inclination angles, on the one hand, the underwater hexapod robot should be able to approach the wall to be climbed, and on the other hand, the underwater hexapod robot should be able to climb up the wall with another inclination angle. Therefore, the switch signals c1(t) and c2(t) are set in the reward function, as follows:

[0087]

[0088] Among them, D is the distance threshold, Θ is the pitch angle threshold, and the value is adjusted according to the training effect.

[0089] In addition, in order to drive the six blade legs evenly and avoid the situation where some legs are stationary for a long time, it is necessary to consider the extreme difference constraint of the blade leg joint angular velocity at each moment to improve the utilization rate of the blade legs. The blade leg utilization characterization function u(t) is defined as They represent the maximum and minimum angular velocities of the six blade leg joints at time t respectively. Q represents the extreme difference threshold of angular velocity, and its value is adjusted according to the training situation.

[0090] The present invention adopts a reinforcement learning method based on DDPG to train walking gaits that adapt to different heading angles and speeds, crossing gaits on walls with different inclination angles, and climbing gaits. The DDPG algorithm is based on the actor-critic framework and is a different-strategy reinforcement learning method. The algorithm has a total of 4 neural networks, including an actor neural network and a critic neural network. The actor neural network and the critic neural network each contain an evaluation network and a target network. The so-called different strategy means that the evaluation strategy actually executed during the training process and the current optimal target strategy are not the same strategy, thereby avoiding the network parameters from being used to calculate gradients while being frequently updated, so as to improve the stability of the training process and make the training easy to converge. The target strategy and its corresponding optimal value function are generated by the target network, and the target network parameters are obtained by soft updating the evaluation network parameters. Therefore, the target network parameters change less, so the target network is used to calculate the gradient. The evaluation strategy is generated by the evaluation network, and a certain amount of noise is added to convert it into a random strategy. It interacts with the environment and updates the evaluation network parameters through the gradient calculated by the target network. New experience is accumulated each time the interaction occurs, thereby ensuring sufficient exploration. The training process of the neural network in the deep reinforcement learning is as follows Figure 5 shown.

[0091] In the specific embodiment of this article, the actor neural network is a four-layer neural network, including two hidden layers, the number of neurons in the hidden layers is 400 and 200 respectively; the structure of the critic neural network is the same as that of the actor neural network. The number of rounds of neural network training is 1000, and each round executes a maximum of 500 sets of actions, and the simulation step length is 0.1s.

[0092] Since the underwater hexapod robot will be affected by the restoring force and restoring torque when operating on walls with different inclination angles, in order to reduce the impact of this effect, the thrust of the propeller is calculated according to the current attitude angle of the underwater hexapod robot to compensate for the restoring force and restoring torque. At the same time, in order to meet the no-slip requirement, the propeller needs to provide sufficient positive pressure. Therefore, based on the above analysis, the total thrust required by the underwater robot is as follows:

[0093]

[0094] Among them, T i (i=1,2,…,8) is the thrust of thruster i; The thrust distribution matrix C t The pseudo-inverse of F gx 、F gy 、F gz and M gx 、M gy 、M gz are the components of the restoring force and restoring moment in the x, y, and z directions respectively; T x 、T y 、T z In order to prevent the blade legs from slipping, the thruster needs to provide thrust components in the x, y, and z directions.

[0095] In this embodiment, the neural network parameters are: playback pool size 20000, batch size 64, discount factor γ = 0.998, soft update coefficient τ = 0.001, actor neural network learning rate α A = 0.00001, critic neural network learning rate α C =0.0001. The values ​​of the parameters in the reward function are as follows: λ1=100, λ2=100, λ3=2, λ4=2, λ5=15, λ6=5, λ7=10, λ8=5. In order to improve the generalization ability of the model, the initial position (x init ,y init ,z init ), initial heading angle ψ init , Expected speed and the wall inclination Set the desired heading angle to ψ d =0rad, the random variables are uniformly distributed, specifically: the initial position of the underwater hexapod robot is (U(-15m,-10m),250m,-99.4m), the initial heading angle is ψ init ~U(-1.41rad,1.41rad), the expected speed is The wall inclination angle is

[0096] Figure 6 This is the reward function change curve of the reinforcement learning algorithm training process in this embodiment. Figure 6 As shown in the figure, the cumulative return of a single round and the average return of each round of the DDPG algorithm can converge and stabilize.

[0097] Figures 7 to 13 The test results of the reinforcement learning model are as follows:

[0098] Figure 7 This is the three-dimensional motion trajectory of the underwater hexapod robot when climbing. Figure 8 (a) and (b) are the horizontal and vertical motion trajectories of the underwater hexapod robot when climbing, respectively. It can be seen from the three-dimensional motion trajectory that the underwater hexapod robot can continuously adjust according to the desired heading angle and climb up the inclined wall. Figure 8 It can be seen that the underwater hexapod robot climbed 16.35m along the positive direction of the y-axis on the slope.

[0099] Fig. 9 (a) and (b) are the heading angle change curve and heading angle error change curve of the underwater hexapod robot, respectively. It can be seen from the results shown in the figure that when the underwater hexapod robot moves on the horizontal plane, it can track the expected heading angle, and the tracking error is basically kept within 0.2 rad.

[0100] Fig.10 (a) and (b) are the pitch angle change curve and pitch angle error change curve of the underwater hexapod robot, respectively. As shown in the figure, before t = 26s, the underwater hexapod robot moves in the horizontal plane. At t = 26s, the pitch angle begins to change and continuously adapts to the inclination of the wall, switching from the horizontal plane to the inclination of The wall takes about 5 seconds to cross, and the pitch angle does not fluctuate much afterwards. The results in the figure show that the underwater hexapod robot can cross between walls with sudden changes in inclination angle.

[0101] Fig.11 (a) and (b) are the forward velocity change curve and forward velocity error change curve of the underwater hexapod robot, respectively. It can be seen from the results that when t=26s, the speed of the underwater hexapod robot drops sharply because the robot reaches the bottom of the wall and starts to cross the wall. After climbing up the wall, the underwater hexapod robot can accelerate to the desired speed in a short time and maintain stability, and the speed error basically does not exceed 0.2m / s.

[0102] The thrust curve of the underwater hexapod robot thruster is as follows: Fig.12 As shown. Fig.12 It can be seen that when the hexapod robot climbs up the wall, the vertical thrust will provide a certain pitch moment, and the main thrust will provide a certain thrust to compensate for the component of the buoyancy force in the body coordinate system.

[0103] Fig.13 (a) to (h) are schematic diagrams of the underwater hexapod robot's motion process in the Gazebo simulation platform. Figures (a) and (c) show the underwater hexapod robot adjusting its heading angle in the horizontal plane, Figures (d) to (f) show the underwater hexapod robot crossing between walls, and Figures (g) and (h) show the underwater hexapod robot climbing along the wall after climbing onto it.

[0104] From the above results, it can be seen that the reinforcement learning algorithm of the present invention can train a model that enables the underwater hexapod robot to achieve constant speed orientation, wall climbing and crossing. The test results of the model also verify the effectiveness of the training.

Claims

1. A method for controlling multi-wall continuous motion of an underwater hexapod robot based on reinforcement learning, wherein the underwater hexapod robot comprises 8 propellers, 6 blade legs and a robot body, wherein 4 propellers are vertically distributed, 4 propellers are horizontally vector-distributed, and the 8 propellers are all embedded in the robot body; the 6 blade legs are semicircular blade legs, and the blade leg hip joints are connected to 6 drive motors on the left and right sides of the robot body, characterized in that The steps include: S1: For a wall with a known inclination angle, use a distance measuring device to obtain the forward distance from the center of mass of the underwater hexapod robot to the wall to be climbed, and at the same time use a distance measuring device to obtain the distance between the center of mass of the underwater hexapod robot and the wall where the underwater hexapod robot is currently located; S2: By analyzing the constant speed directional climbing and wall crossing tasks, the state space, action space and reward function for the smooth crossing and constant speed directional climbing tasks are set; the directional constant speed means that the robot can move at the expected forward speed and the expected heading angle, and the smooth crossing means that the robot can achieve continuous movement between the sudden inclination walls while maintaining stability; S3: Using the deep deterministic policy gradient reinforcement learning method, we train an underwater hexapod robot to climb along walls of different inclinations at a constant speed and in a directional manner, and to smoothly cross between walls of different inclinations. S4: In order to improve the stability of the underwater hexapod robot's movement, on the one hand, a basic thrust is applied to the underwater hexapod robot's propeller to ensure that the blade legs of the underwater hexapod robot do not slip during movement. On the other hand, in order to reduce the influence of the restoring force and restoring torque on the movement of the underwater hexapod robot, additional thrust is applied to the underwater hexapod robot's propeller according to the underwater hexapod robot's current attitude angle to overcome the restoring force and restoring torque.

2. The method for controlling multi-wall continuous motion of an underwater hexapod robot based on reinforcement learning according to claim 1, characterized in that: In S1, for a wall with a known inclination angle, a distance measuring device fixed to the mass center of the underwater hexapod robot is used to obtain the forward distance d from the mass center of the underwater hexapod robot to the wall to be climbed, and a Doppler velocimeter fixed to the mass center of the underwater hexapod robot is used to obtain the height h of the mass center of the underwater hexapod robot relative to the wall where the robot is currently located. Whether the blade leg touches the ground is determined by the height h and the joint angle q of each blade leg of the underwater hexapod robot. i The judgment is as follows: Among them, a i represents the ground contact state of the i-th blade leg, a i =1 indicates that the i-th blade leg touches the ground, a i =0 indicates that the i-th blade leg has not touched the ground, i is the number of the blade leg, i=1,2,…,6; l r is the blade leg radius; θ are the inclination angle of the wall where the i-th blade leg of the underwater robot is located and the pitch angle of the underwater hexapod robot; b x pi and b y pi They respectively represent the coordinate components of the landing point of the underwater hexapod robot along the x and y directions in the body coordinate system.

3. The method for controlling multi-wall continuous motion of an underwater hexapod robot based on reinforcement learning according to claim 1, characterized in that: In step S2, by analyzing the constant speed directional climbing and wall crossing tasks, the state space, action space and reward functions for the smooth crossing and constant speed directional climbing tasks are set; the specific steps are as follows: Considering the task requirements and robot characteristics, a 15-dimensional column vector is selected as the state space S, that is, Where T is the matrix transpose symbol; v x are the expected forward velocity and actual forward velocity of the underwater hexapod robot, respectively; ψ d , ψ are the expected heading angle and the actual heading angle of the underwater hexapod robot, respectively; θ are the inclination angle of the wall to be climbed and the pitch angle of the underwater hexapod robot; is the rolling angle of the underwater hexapod robot; the action set is defined as in is the joint angular velocity of the i-th blade leg; In order to enable the underwater hexapod robot to achieve constant speed and directional climbing, and to smoothly cross between walls with different inclination angles, the reward function is set as: Among them, λ1,λ2,…,λ8 are parameters to be adjusted that are greater than zero, t represents the current time, s t is the state quantity at the current moment, a t is the action executed at the current moment, g(t) is the characterization function of the underwater hexapod robot's ground contact stability, c1(t) is the switch signal of whether to start crossing the wall, c2(t) is the switch signal of whether to climb up the wall, and u(t) is the characterization function of the blade leg utilization rate; the first four items of the reward function are set according to the error of the state variables in the state space to achieve constant speed orientation and posture stability; the last four items are functions of the state quantity and action quantity, which are used to guide the robot to learn to cross stably, and are specifically defined as follows: In order to make the support area formed by the support legs greater than zero, the number of support legs of the underwater hexapod robot needs to be greater than or equal to 3, and there must be support legs on both sides of the underwater hexapod robot body. Therefore, the ground contact stability characterization function g is defined as: in, In order to achieve the smooth crossing of the underwater hexapod robot between walls with different inclination angles, on the one hand, the underwater hexapod robot should be able to approach the wall to be climbed, and on the other hand, the underwater hexapod robot should be able to climb up the wall of another inclination angle. Therefore, the switch signals c1(t) and c2(t) are set in the reward function, as follows: Where D is the distance threshold, Θ is the pitch angle threshold, and the value is adjusted according to the training effect; In addition, in order to drive the six blade legs evenly and avoid the situation where individual legs are stationary for a long time, it is necessary to consider the extreme difference constraint of the blade leg joint angular velocity at each moment to improve the utilization rate of the blade legs. The blade leg utilization characterization function u(t) is defined as: in They represent the maximum and minimum angular velocities of the six blade leg joints at time t respectively. Q represents the extreme difference threshold of angular velocity, and its value is adjusted according to the training situation.

4. The method for controlling multi-wall continuous motion of an underwater hexapod robot based on reinforcement learning according to claim 1, characterized in that: In the step S3, a reinforcement learning method of deep deterministic policy gradient is used to train a gait that enables the underwater hexapod robot to climb along walls of different inclinations at a constant speed and in a directional manner and to smoothly cross between walls of different inclinations; The DDPG algorithm is based on an actor-critic framework, including an actor neural network and a critic neural network. The actor neural network and the critic neural network each contain an evaluation network and a target network. The so-called different strategies refer to that the evaluation strategy actually executed during the training process is not the same as the current optimal target strategy, so as to avoid the network parameters being used to calculate gradients while being frequently updated, so as to improve the stability of the training process and make the training easy to converge. The target strategy and its corresponding optimal value function are generated by the target network, and the target network parameters are obtained by soft updating of the evaluation network parameters. Therefore, the target network parameters change less, so the target network is used to calculate the gradient. The evaluation strategy is generated by the evaluation network, and noise is added to convert it into a random strategy. The evaluation network parameters are updated through the gradient calculated by the target network to interact with the environment. Each interaction accumulates new experience to ensure sufficient exploration.

5. The method for controlling multi-wall continuous motion of an underwater hexapod robot based on reinforcement learning according to claim 1, characterized in that: In the step S4, in order to improve the stability of the underwater hexapod robot's movement, on the one hand, a basic thrust is applied to the underwater hexapod robot's propeller to ensure that the blade legs of the underwater hexapod robot do not slip during movement; on the other hand, in order to reduce the influence of the restoring force and the restoring torque on the movement of the underwater hexapod robot, an additional thrust is applied to the underwater hexapod robot's propeller according to the current attitude angle of the underwater hexapod robot to overcome the restoring force and the restoring torque; According to the blade leg joint torque τ of the generated gait i , we can get: Among them, F τix 、F τiy are the components of the force acting on the touchdown point by the blade leg joint torque along the x and y directions in the touchdown point coordinate system, respectively. r is the blade leg radius; According to the no-slip constraint, the vertical force F required for the blade leg to touch the point is calculated. Tiy , the specific calculation formula is as follows: Where μ is the ground friction coefficient; N i is the ground support force at the touchdown point; According to the DH coordinate transformation, the vertical force F required for the blade leg to touch the ground is Tiy The force required to convert to the center of mass of the robot is provided by the thruster. The thrust components of the thruster in the three directions of the body coordinate system are recorded as T x , T y , T z , then F Tiy and T x , T y , T z Satisfies the following relationship: Therefore, the thruster base thrust for: in, is the basic thrust value of the i-th thruster; C t is the thrust distribution matrix; The restoring force and restoring torque of the underwater hexapod robot are determined by gravity G, buoyancy B and attitude angle. The components of the restoring force in the x, y and z directions are denoted as F gx 、F gy 、F gz The components of the restoring moment in the x, y, and z directions are denoted as M gx 、M gy 、M gz , then the six components are specifically expressed as: Among them, x c ,y c ,z c is the coordinate of the buoyancy center of the underwater robot in the body coordinate system; Therefore, the additional thrust of the propeller for: in, is the additional thrust value of the i-th thruster; C t is the thrust distribution matrix; In summary, the total thrust value of the i-th thruster is

Images