Underwater hexapod robot multi-wall continuous motion control method based on reinforcement learning

Abstract

The application provides a kind of underwater six-legged robot multi-wall continuous motion control method based on reinforcement learning, utilizes ranging equipment to obtain the forward distance from the centroid of underwater six-legged robot to the wall surface to be climbed, simultaneously utilizes ranging equipment to obtain the distance between the centroid of underwater six-legged robot and the wall surface currently occupied by underwater six-legged robot;Through the analysis of constant speed and directional climbing and wall surface crossing task, set state space, action space and reward function for smooth crossing and constant speed and directional climbing task;Using deep deterministic policy gradient reinforcement learning method, the gait of smooth crossing between different inclination wall surfaces is trained.The application avoids the problems of complex modeling and unstable gait switching in traditional gait planning method, so that underwater six-legged robot can cross between different inclination wall surfaces and climb along different inclination wall surfaces at constant speed and direction, and the propeller thrust compatible with the torque of blade leg can be solved to ensure the motion stability of underwater six-legged robot.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater hexapod robots, in particular to a wall climbing and crossing gait planning and control method for underwater hexapod robots. BACKGROUND

[0002] In recent years, autonomous underwater vehicles (AUVs) and remotely operated vehicles (ROVs) have been widely used in ocean exploration, development and utilization. However, the traditional AUVs and ROVs are driven by propellers, which cannot interact with the environment, i.e. they are not suitable for performing local fine work tasks. In order to improve the work ability of marine equipment, propeller and leg-foot hybrid driving has become a solution.

[0003] A precise mathematical model can fully describe the motion of a leg-foot robot, such as a spring-loaded inverted pendulum model. However, the model-based gait planning method is computationally complex and generally has unmodeled dynamics. The gait planning method based on bionics can effectively avoid the above problems. With the continuous development of the central pattern generator (CPG) theory, many different oscillator models and CPG networks have been applied to gait planning. However, the existing CPG-based methods mostly generate periodic gaits, which greatly limit the mobility of leg-foot robots. How to design a non-periodic mapping to generate a more flexible and autonomous gait is a very challenging task. The core of reinforcement learning is to learn from interaction for goal-oriented learning, which is used to solve sequential decision-making problems. Continuous motion on different walls can be classified as a sequential decision-making problem, i.e. how to coordinate the control commands of the legs and the propeller, so that the robot can smoothly transition and climb on different inclined walls while maintaining the required speed and yaw angle. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the present application provides a multi-wall continuous motion control method for underwater hexapod robots based on reinforcement learning. The present application designs an adaptive gait planning and control method based on reinforcement learning. The present application does not rely on a model, avoiding complex calculations and gait switching, and autonomously generates and adjusts the gait during robot motion. At the same time, through certain force analysis, the propeller thrust suitable for the joint torque of the knife-edge leg is solved, so that the underwater hexapod robot can smoothly switch between different inclined walls and stably climb along the inclined wall.

[0005] The technical scheme adopted by the present application to solve its technical problems is:

[0006] The application provides a wall climbing and crossing gait planning and control method for an underwater six-legged robot based on reinforcement learning.

[0007] S1: for a wall surface with a known inclination angle, a ranging device is used to obtain the forward distance from the centroid of the underwater six-legged robot to the wall surface to be climbed, and the ranging device is used to obtain the distance from the centroid of the underwater six-legged robot to the wall surface currently occupied by the underwater six-legged robot;

[0008] S2: by analyzing the constant speed and directional climbing and wall crossing task, a state space, an action space and a reward function for smooth crossing and constant speed and directional climbing task are set; the directional and constant speed means that the robot can move at an expected forward speed and an expected heading angle, and the smooth crossing means that the robot can continuously move between walls with sudden inclination under the premise of keeping stable;

[0009] S3: a reinforcement learning method of deep deterministic policy gradient (DDPG) is used to train a gait for the underwater six-legged robot to climb at a constant speed and direction along different inclination walls and to smoothly cross between different inclination walls;

[0010] S4: in order to improve the stability of the underwater six-legged robot, on the one hand, a basic thrust is applied to the propeller of the underwater six-legged robot to ensure that the blade leg does not slip when the underwater six-legged robot moves, and on the other hand, in order to reduce the influence of the restoring force and the restoring torque on the movement of the underwater six-legged robot, an additional thrust is applied to the propeller of the underwater six-legged robot according to the current attitude angle of the underwater six-legged robot to overcome the restoring force and the restoring torque.

[0011] In S1, for a wall surface with a known inclination angle, a ranging device fixed to the centroid of the underwater six-legged robot is used to obtain the forward distance d from the centroid of the underwater six-legged robot to the wall surface to be climbed, and a Doppler velocimeter fixed to the centroid of the underwater six-legged robot is used to obtain the height h of the centroid of the underwater six-legged robot relative to the wall surface currently occupied by the robot, and whether the blade leg touches the ground is determined by the height h and the joint angle q i of each blade leg of the underwater six-legged robot.

[0012]

[0013] wherein a i represents the ground contact state of the i-th blade leg, a i = 1 indicates that the i-th blade leg touches the ground, and a i= 0 indicates that the i-th blade leg does not touch the ground, i is the number of blade legs, i = 1, 2, …, 6; l r is the radius of the blade leg; θ are the inclination angle of the wall surface where the underwater robot is located and the pitch angle of the underwater hexapod robot, respectively; b x pi and b y pi respectively represent the coordinate components of the underwater hexapod robot's touch point in the body coordinate system along the x and y directions.

[0014] In the step S2, by analyzing the constant speed and directional climbing and wall crossing tasks, the state space, action space and reward function oriented to the smooth crossing and constant speed and directional climbing tasks are set; the specific steps are as follows:

[0015] Considering the task requirements and the characteristics of the robot, a 15-dimensional column vector is selected as the state space S, i.e. Where T is the matrix transpose symbol; v x are the expected forward speed and the real forward speed of the underwater hexapod robot, respectively; ψ d , ψ are the expected heading angle and the real heading angle of the underwater hexapod robot, respectively; θ are the inclination angle of the wall surface to be climbed and the pitch angle of the underwater hexapod robot, respectively; is the roll angle of the underwater hexapod robot; the action set is defined as Where is the joint angular velocity of the i-th blade leg;

[0016] In order to make the underwater hexapod robot achieve constant speed and directional climbing, and smoothly cross different inclination walls, the reward function is set as:

[0017]

[0018] Where λ1, λ2, …, λ8 are greater than zero to be adjusted parameters, t represents the current time, s t is the state quantity at the current time, a t is the action executed at the current time, g(t) is the underwater hexapod robot touch stability representation function, c1(t) is the switch signal whether to start crossing the wall, c2(t) is the switch signal whether to climb the wall, u(t) is the blade leg utilization rate representation function; the first four items of the reward function are set according to the error of the state variable in the state space, in order to achieve constant speed and directional and posture stability; the last four items are functions of state and action, which are used to guide the robot to learn to cross stably, and their specific definitions are as follows:

[0019] In order to make the support leg constitute the support area greater than zero, the number of support legs of the underwater six-legged robot needs to be greater than or equal to 3, and both sides of the body of the underwater six-legged robot need to have support legs, so the touch ground stability representation function g is defined as:

[0020]

[0021] Wherein,

[0022] In order to realize the smooth crossing of the underwater six-legged robot between different inclined walls, on the one hand, the underwater six-legged robot needs to be able to approach the wall to be climbed, and on the other hand, the underwater six-legged robot needs to be able to climb the wall with another inclination, so the switch signals c1(t) and c2(t) are set in the reward function, as follows:

[0023]

[0024]

[0025] Wherein, D is the distance threshold, and Θ is the pitch angle threshold, which is adjusted according to the training effect.

[0026] In addition, in order to make the six knife legs drive uniformly and avoid the situation that individual legs are stationary for a long time, the range constraint of the joint angular velocity of each knife leg at each moment needs to be considered to improve the utilization rate of the knife leg, and the knife leg utilization rate representation function u(t) is defined as:

[0027]

[0028] Wherein Respectively represent the maximum and minimum values of the joint angular velocity of the six knife legs at time t, and Q represents the angular velocity range threshold, which is adjusted according to the training condition.

[0029] In the step S3, a reinforcement learning method using deep deterministic policy gradient (DDPG) is used to train a gait that can make the underwater six-legged robot realize constant speed and directional climbing along different inclined walls and can smoothly cross between different inclined walls;

[0030] The DDPG algorithm, based on the actor-critic framework, is a hetero-policy reinforcement learning method comprising an actor neural network and a critic neural network. Each neural network includes an evaluation network and a target network. Heterogeneous policies refer to the fact that the evaluation policy actually implemented during training is not the same as the current optimal target policy. This avoids using network parameters for gradient calculation while they are frequently updated, thus improving the stability of the training process and making it easier to converge. The target policy and its corresponding optimal value function are generated by the target network, whose parameters are obtained through soft updates of the evaluation network parameters. Therefore, the target network parameters change relatively little, and the gradient is calculated using the target network. The evaluation policy is generated by the evaluation network, and noise is added to transform it into a stochastic policy, which interacts with the environment and updates the evaluation network parameters using the gradient calculated by the target network. Each interaction accumulates new experience, ensuring sufficient exploration.

[0031] In step S4, in order to improve the stability of the underwater hexapod robot's movement, on the one hand, a basic thrust is applied to the underwater hexapod robot's thrusters to ensure that the blade legs of the underwater hexapod robot do not slip during movement; on the other hand, in order to reduce the influence of restoring force and restoring torque on the underwater hexapod robot's movement, an additional thrust is applied to the underwater hexapod robot's thrusters based on the current attitude angle of the underwater hexapod robot to overcome the restoring force and restoring torque.

[0032] Based on the blade leg joint torque τ of the generated gait i We can obtain:

[0033]

[0034] Among them, F τix F τiy These are the components of the force exerted by the blade leg joint torque on the contact point along the x and y directions in the contact point coordinate system, respectively. r The radius of the blade leg;

[0035] The vertical force F required at the contact point of the blade leg is determined based on the non-slip constraint. Tiy The specific calculation formula is as follows:

[0036]

[0037] Where μ is the ground friction coefficient; N i The ground support force at the point of contact;

[0038] Based on the DH coordinate transformation, the vertical force F required at the contact point of the blade leg is... Tiy The force required to transfer to the robot's center of mass is provided by the thrusters. The components of the thruster thrust along the three directions of the body coordinate system are denoted as T. x Ty , T z , F Tiy and T x , T y , T z satisfy the following relationship:

[0039]

[0040] Therefore, the total thrust of the i-th propeller is :

[0041]

[0042] wherein, is the basic thrust value of the i-th propeller; C t is the thrust distribution matrix;

[0043] The restoring force and restoring moment of the underwater hexapod robot are determined by the gravity G, buoyancy B and attitude angle, and the components of the restoring force in the x, y and z directions are respectively denoted as F gx , F gy , F gz , and the components of the restoring moment in the x, y and z directions are respectively denoted as M gx , M gy , M gz , and the six components are specifically represented as:

[0044]

[0045] wherein, x c , y c , z c is the coordinate of the underwater robot's buoyancy center in the body coordinate system;

[0046] Therefore, the additional thrust of the propeller is :

[0047]

[0048] wherein, is the additional thrust value of the i-th propeller; C t is the thrust distribution matrix;

[0049] Therefore, the total thrust of the i-th propeller is

[0050] The beneficial effects of the present application mainly embody in the following two aspects:

[0051] (1) For the multi-task operation scenario of underwater hexapod robot, a reward function is designed, which makes the trained gait consider directional and constant speed, wall climbing and wall crossing, so as to avoid the problems of complex modeling and unstable gait switching in traditional gait planning method.

[0052] (2) For the hybrid driven underwater hexapod robot, a gait planning method based on reinforcement learning is proposed, which enables the underwater hexapod robot to cross different inclined walls and climb along different inclined walls at a constant speed, and at the same time, it can solve the propeller thrust compatible with the torque of the blade leg, ensuring the stability of the underwater hexapod robot. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 Fig. 1 is a schematic diagram of the wall climbing and crossing gait planning and control method of the underwater hexapod robot based on reinforcement learning.

[0054] Figure 2 Fig. 2 (a) is a schematic diagram of the underwater hexapod robot under the Gazebo simulation platform, Figure 2 Fig. 2 (b) is a schematic diagram of the inclined wall under the Gazebo simulation platform.

[0055] Figure 3 Fig. 3 is a schematic diagram of the position of the six hip joints of the underwater hexapod robot.

[0056] Figure 4 Fig. 4 is a schematic diagram of the position of the six propellers of the underwater hexapod robot.

[0057] Figure 5 Fig. 5 is a schematic diagram of the neural network training process in deep reinforcement learning.

[0058] Figure 6 Fig. 6 is a reward function curve in the reinforcement learning algorithm.

[0059] Figure 7 Fig. 7 is a three-dimensional motion trajectory diagram of the underwater hexapod robot when climbing.

[0060] Figure 8 Fig. 8 (a) and (b) are respectively the horizontal plane motion trajectory diagram and the vertical plane motion trajectory diagram of the underwater hexapod robot when climbing.

[0061] Figure 9 Fig. 9 (a) and (b) are respectively the heading angle change curve and the heading angle error change curve of the underwater hexapod robot.

[0062] Figure 10 Fig. 10 (a) and (b) are respectively the pitch angle change curve and the pitch angle error change curve of the underwater hexapod robot.

[0063] Figure 11(a) and (b) are respectively the forward velocity curve and the forward velocity error curve of the underwater hexapod robot.

[0064] Figure 12 is a propeller thrust curve of the underwater hexapod robot.

[0065] Figure 13 (a) to Figure 13 (h) is a schematic diagram of the motion process of the underwater hexapod robot in the Gazebo simulation platform. DETAILED DESCRIPTION

[0066] The application will be further described below in combination with the drawings and examples.

[0067] The application proposes a gait planning and control method for an underwater hexapod robot driven by a propeller and a semicircular blade leg based on reinforcement learning, which adjusts the joint angle of the blade leg and the propeller thrust through self-adaption, so that the underwater hexapod robot can complete the task of crossing different inclined walls and can climb at a constant speed and in a constant direction on different inclined walls. The method comprises the following steps: S1: for a wall with a known inclination, a ranging device is used to obtain the forward distance from the center of mass of the underwater hexapod robot to the wall to be climbed, and the ranging device is used to obtain the distance from the center of mass of the underwater hexapod robot to the wall currently located by the underwater hexapod robot; S2: by analyzing the task of climbing at a constant speed and in a constant direction and the task of crossing the wall, a state space, an action space and a reward function facing the tasks of smooth crossing and climbing at a constant speed and in a constant direction are set; S3: a reinforcement learning method of deep deterministic policy gradient (DDPG) is used to train a gait that can enable the underwater hexapod robot to climb at a constant speed and in a constant direction along different inclined walls and to smoothly cross different inclined walls; and S4: in order to improve the stability of the motion of the underwater hexapod robot, on the one hand, a basic thrust is applied to the propeller of the underwater hexapod robot to ensure that the blade leg of the underwater hexapod robot does not slip during motion, and on the other hand, in order to reduce the influence of the restoring force and the restoring torque on the motion of the underwater hexapod robot, an additional thrust is applied to the propeller of the underwater hexapod robot according to the current attitude angle of the underwater hexapod robot to overcome the restoring force and the restoring torque. The gait planning and control method proposed in the application can enable the robot to cross and climb different inclined walls by adaptively adjusting the gait and the propeller thrust.

[0068] Therefore, the application designs a gait planning and control method based on reinforcement learning, which is not based on a model, avoids complex calculation and gait switching, and solves the propeller thrust suitable for the torque of the blade leg joint through certain force analysis, so that the underwater hexapod robot can smoothly switch and stably climb along the inclined wall.

[0069] In view of the problems of complex modeling and poor adaptability of existing gait planning methods, the DDPG reinforcement learning algorithm is used to propose an adaptive gait planning and control method for wall climbing and crossing tasks. When designing the gait planning and control method for wall climbing and crossing, the following difficulties exist: 1. For multiple tasks such as constant speed and direction climbing and wall crossing, how to set the reward function to accurately reflect the task requirements; 2. For continuous control problems with high-dimensional action space, how to improve the training efficiency; 3. How to improve the adaptability of gait planning, so that the underwater hexapod robot can smoothly cross and climb different walls.

[0070] The present application will be further described in conjunction with embodiments, drawings:

[0071] Figure 1 It is a schematic diagram of a wall climbing and crossing gait planning and control method for an underwater hexapod robot based on reinforcement learning. First, set the expected climbing speed v of the underwater hexapod robot xd And the expected heading angle ψ d And use the current climbing speed v of the underwater hexapod robot x And the heading angle ψ to obtain the heading angle error and the climbing speed error, and at the same time, based on the state of each knife leg touching the ground, a stability function is constructed for the underwater hexapod robot to climb stably. Second, construct a reward function r t Containing heading speed error, attitude angle constraint and stability representation, wherein the heading speed error ensures that the underwater hexapod robot walks at the expected heading speed; the attitude angle constraint and the distance constraint from the wall ensure that the underwater hexapod robot can approach and climb the wall; the stability representation ensures that the underwater hexapod robot touches the ground and walks stably; the joint angular velocity extreme difference constraint of the knife leg ensures that each knife leg is uniformly driven, thereby avoiding the situation that an individual leg is stationary for a long time. Finally, based on the DDPG reinforcement learning method, the target neural network is trained to obtain the mapping relationship between the state quantities climbing speed v d , heading angle ψ, pitch angle θ, six leg rotation angles q1,…,q6, distance measurement value d and six leg angular velocity When the underwater hexapod robot crosses the wall with a tilt angle of xd And stably climbs, the mapping relationship between the state quantities climbing speed v x , heading angle ψ, pitch angle θ, six leg rotation angles q1,…,q6, distance measurement value d and six leg angular velocity .

[0072] Figure 2 (a) and (b) are respectively the underwater hexapod robot and the inclined wall schematic diagram under the Gazebo simulation platform. Under the Gazebo simulation platform, the model of the underwater hexapod robot and the inclined wall is constructed for simulation. Figure 3 It is a schematic diagram of the hip joint position of the six legs of the hybrid driven underwater hexapod robot. Figure 4Schematic diagram of six propeller positions of a hybrid-driven underwater hexapod robot.

[0073] Since the kinematic model of the underwater hexapod robot is relatively complex, and the gait planning method based on CPG can only generate periodic or approximately periodic gaits, the existing model-based optimization or random search methods and the gait planning method based on CPG are difficult to be used in the operation condition of crossing different inclination walls. Therefore, the present application proposes to use a deep reinforcement learning algorithm to plan the crossing and climbing gaits of the hexapod robot, which specifically includes the following steps:

[0074] S1: For the wall with a known inclination, a ranging device is used to obtain the forward distance from the center of mass of the underwater hexapod robot to the wall to be climbed, and a ranging device is used to obtain the distance from the center of mass of the underwater hexapod robot to the wall currently occupied by the underwater hexapod robot;

[0075] S2: By analyzing the constant speed and directional climbing and wall crossing tasks, a state space, an action space and a reward function for smooth crossing and constant speed and directional climbing tasks are set;

[0076] S3: A reinforcement learning method based on DDPG is used to train a gait that can enable the underwater hexapod robot to achieve constant speed and directional climbing along different inclination walls and smooth crossing between different inclination walls;

[0077] S4: In order to improve the stability of the motion of the underwater hexapod robot, on the one hand, a basic thrust is applied to the propeller of the underwater hexapod robot to ensure that the knife leg of the underwater hexapod robot does not slip during motion, and on the other hand, in order to reduce the influence of the restoring force and the restoring torque on the motion of the underwater hexapod robot, an additional thrust is applied to the propeller of the underwater hexapod robot according to the current attitude angle of the underwater hexapod robot to overcome the restoring force and the restoring torque.

[0078] For the crossing problem of the underwater hexapod robot between different inclination walls, the present application uses a range finder installed at the center of mass to measure the forward distance from the center of mass of the underwater hexapod robot to the inclined wall, and uses a Doppler velocity meter fixed to the center of mass of the underwater hexapod robot to obtain the height of the center of mass of the underwater hexapod robot relative to the wall currently occupied by the robot, and sets a threshold to determine whether to start crossing the wall, and introduces the signal into the reward function, and simultaneously uses the prior known wall inclination information to learn the mapping relationship between the underwater hexapod robot smoothly crossing different inclination walls, directional and constant speed climbing walls, wall inclination and the distance between the robot center of mass and the wall by using the reinforcement learning algorithm of DDPG.

[0079] In order to make the support area formed by the support legs greater than zero, the number of support legs of the underwater hexapod robot needs to be greater than or equal to 3, and the two sides of the body of the underwater hexapod robot need to have support legs. The expression of whether a stable support domain can be formed is:

[0080]

[0081] wherein, a i represents the ground contact state of the ith blade leg, a i = 1 indicates that the ith blade leg is in contact with the ground, a i = 0 indicates that the ith blade leg is not in contact with the ground, i is the number of the blade leg, i = 1, 2, …, 6.

[0082] Considering the task requirements and the characteristics of the robot, a 15-dimensional column vector is selected as the state space S, that is, wherein T is the matrix transpose symbol; q i is the joint angle of the ith blade leg of the underwater hexapod robot; d is the distance measurement value of the distance measuring device at the centroid of the underwater hexapod robot, that is, the distance between the underwater robot and the wall surface to be climbed; h is the height of the centroid of the underwater hexapod robot relative to the wall surface currently occupied by the underwater hexapod robot; v x are the expected forward speed and the actual forward speed of the underwater hexapod robot, respectively; ψ d , ψ are the expected heading angle and the actual heading angle of the underwater hexapod robot, respectively; θ are the inclination angle of the wall surface to be climbed and the pitch angle of the underwater hexapod robot, respectively; is the roll angle of the underwater hexapod robot. The action set is defined as wherein is the joint angular velocity of the ith blade leg.

[0083] In order to enable the underwater hexapod robot to achieve constant speed and directional climbing, and to smoothly cross different inclination wall surfaces, the reward function is set as

[0084]

[0085] wherein λ1…λ8 are coefficients greater than zero, t represents the current time, s t is the state quantity at the current time, a t is the action performed at the current time, g(t) is a stability representation function of the underwater hexapod robot in contact with the ground, c1(t) is a switch signal indicating whether to start crossing the wall surface, c2(t) is a switch signal indicating whether to climb the wall surface, and u(t) is a utilization rate representation function of the blade leg. The first four terms of the reward function are set according to the error of the state variable in the state space, so as to achieve constant speed and directional climbing and posture stability. The last four terms are functions of the state quantity and the action quantity, which are used to guide the robot to learn to cross stably, and the specific definitions are as follows.

[0086] In order to realize the smooth crossing of the underwater six-legged robot between different inclination walls, on the one hand, the underwater six-legged robot can approach the wall to be climbed, and on the other hand, the underwater six-legged robot can climb the wall with another inclination, therefore, the switch signals c1(t) and c2(t) are set in the reward function, as follows

[0087]

[0088] Wherein, D is a distance threshold, and Θ is a pitch angle threshold, and the values are adjusted according to the training effect.

[0089] In addition, in order to uniformly drive the six blade legs and avoid the situation that individual legs are stationary for a long time, the range constraint of the joint angular velocity of each blade leg at each moment is considered, and the utilization rate of the blade leg is improved. The utilization rate of the blade leg is represented by a function u(t), wherein The maximum and minimum values of the joint angular velocity of the six blade legs at t are represented by max and min respectively, and Q represents the angular velocity range threshold, and the value is adjusted according to the training condition.

[0090] The underwater six-legged robot is trained to adapt to different walking gaits, different inclination wall crossing gaits and climbing gaits by using the DDPG-based reinforcement learning method. The DDPG algorithm is based on the actor-critic framework and is a heterogeneous policy reinforcement learning method. The algorithm has four neural networks, including an actor neural network and a critic neural network, and the actor neural network and the critic neural network each include an evaluation network and a target network. The so-called heterogeneous policy means that the evaluation strategy actually executed in the training process and the current optimal target strategy are not the same strategy, so as to avoid that the network parameters are used to calculate the gradient at the same time of frequent updating, so as to improve the stability of the training process and make the training easy to converge. The target strategy and the corresponding optimal value function are generated by the target network, and the target network parameters are obtained by soft updating the evaluation network parameters, so the target network parameters change little, and the gradient is calculated by using the target network. The evaluation strategy is generated by the evaluation network, and a certain noise is added to convert it into a random strategy, which is interacted with the environment and the gradient calculated by the target network is used to update the evaluation network parameters, and new experiences are accumulated every time, so as to ensure sufficient exploration. The training process of the neural network in the deep reinforcement learning is as shown in Figure 5 .

[0091] In the specific embodiments of the present application, the actor neural network is a four-layer neural network, including two hidden layers, and the number of hidden layer neurons is 400 and 200 respectively; the critic neural network has the same structure as the actor neural network. The number of rounds of neural network training is 1000, and a maximum of 500 actions are executed every round, and the simulation step is 0.1s.

[0092] Because the underwater hexapod robot is subject to restoring forces and restoring torques when operating on walls with varying inclination angles, the thruster thrust is calculated to compensate for these forces and torques based on the robot's current attitude angle. Simultaneously, to meet the no-slip requirement, the thrusters must provide sufficient normal force. Therefore, based on the above analysis, the total thrust required by the underwater robot is as follows:

[0093]

[0094] Among them, T i (i = 1, 2, ..., 8) represents the thrust of thruster i; Assigning the thrust matrix C t The pseudo-inverse; F gx F gy F gz and M gx M gy M gz These are the components of the restoring force and restoring torque in the x, y, and z directions, respectively; T x T y T z To prevent the blade legs from slipping, the thruster needs to provide thrust components in the x, y, and z directions.

[0095] In this embodiment, the neural network parameters are set as follows: playback pool size 20000, batch size 64, discount factor γ = 0.998, soft update coefficient τ = 0.001, and actor neural network learning rate α. A =0.00001, the learning rate α of the commentator's neural network C =0.0001. The parameters in the reward function take the following values: λ1=100, λ2=100, λ3=2, λ4=2, λ5=15, λ6=5, λ7=10, λ8=5. To improve the generalization ability of the model, the initial position (x) of the underwater hexapod robot is randomly generated before the start of each round during training. init ,y init ,z init ), initial heading angle ψ init Expected speed and wall inclination angle Set the desired heading angle as ψ d =0 rad, and all random variables are uniformly distributed. Specifically, the initial position of the underwater hexapod robot is (U(-15m,-10m),250m,-99.4m), and the initial heading angle is ψ. init ~U(-1.41rad, 1.41rad), desired speed is The wall inclination angle is

[0096] Figure 6 This is the curve showing the change of the reward function during the training process of the reinforcement learning algorithm in this embodiment. Figure 6 As shown, the cumulative reward of a single round and the average reward of each round of the DDPG algorithm can converge and tend to stabilize.

[0097] Figures 7 to 13 The test results for the reinforcement learning model are as follows:

[0098] Figure 7 This is a 3D motion trajectory diagram of an underwater hexapod robot climbing. Figure 8 (a) and (b) show the horizontal and vertical motion trajectories of the underwater hexapod robot during climbing, respectively. The 3D motion trajectory diagrams demonstrate that the underwater hexapod robot can continuously adjust its course according to the desired heading angle and climb an inclined wall. Figure 8 It can be seen that the underwater hexapod robot climbed 16.35m along the positive y-axis on the inclined plane.

[0099] Figure 9 (a) and (b) show the heading angle variation curve and heading angle error variation curve of the underwater hexapod robot, respectively. As can be seen from the figures, the underwater hexapod robot can track the desired heading angle when moving on the horizontal plane, and the tracking error is basically kept within 0.2 rad.

[0100] Figure 10 (a) and (b) show the pitch angle variation curve and pitch angle error variation curve of the underwater hexapod robot, respectively. As shown in the figure, before t=26s, the underwater hexapod robot moves in the horizontal plane. At t=26s, the pitch angle begins to change and continuously adapts to the wall tilt angle, switching from the horizontal plane to a tilt angle of 0. The wall was traversed in approximately 5 seconds, after which the pitch angle remained relatively stable. The illustrated results demonstrate that the underwater hexapod robot is capable of crossing walls with abrupt changes in tilt angle.

[0101] Figure 11 (a) and (b) show the forward velocity variation curve and forward velocity error variation curve of the underwater hexapod robot, respectively. As can be seen from the results, when t = 26s, the speed of the underwater hexapod robot drops sharply as it reaches the bottom of the wall and begins to cross it. After crossing and climbing up the wall, the underwater hexapod robot can accelerate to the desired speed in a short time and maintain stability, with the speed error basically not exceeding 0.2m / s.

[0102] The thrust variation curve of the underwater hexapod robot's thruster is as follows: Figure 12 As shown. By Figure 12 It can be seen that when the hexapod robot climbs onto the wall, the vertical thrust provides a certain pitching moment, and the main thrust provides a certain thrust to compensate for the component of buoyancy in the body coordinate system.

[0103] Figure 13 (a) to (h) are schematic diagrams of the movement process of the underwater hexapod robot in the Gazebo simulation platform. Figure (a), figure (c) are the stage of adjusting the heading angle of the underwater hexapod robot in the horizontal plane, figure (d) to figure (f) are the stage of crossing the wall surface of the underwater hexapod robot, figure (g) and figure (h) are the stage of climbing along the wall surface after the underwater hexapod robot climbs up the wall surface.

[0104] From the above results, it can be known that the reinforcement learning algorithm of the present application can train a model to enable the underwater hexapod robot to realize constant speed and orientation, wall surface climbing and crossing. The test results of the model also verify the effectiveness of the training.

Claims

1. A method for continuous motion control of an underwater hexapod robot on multiple walls based on reinforcement learning, wherein the underwater hexapod robot comprises 8 thrusters, 6 blade-shaped legs, and a robot body, wherein 4 thrusters are vertically distributed, 4 thrusters are horizontally vector distributed, and all 8 thrusters are embedded in the robot body; the 6 blade-shaped legs are semi-circular arc-shaped, and the hip joints of the blade-shaped legs are connected to 6 drive motors on the left and right sides of the robot body, characterized in that... Includes the following steps: S1: For a wall with a known inclination angle, use a ranging device to obtain the forward distance from the center of mass of the underwater hexapod robot to the wall to be climbed, and at the same time use a ranging device to obtain the distance between the center of mass of the underwater hexapod robot and the wall where the underwater hexapod robot is currently located. S2: By analyzing constant-speed orientation climbing and wall crossing tasks, we set the state space, action space, and reward function for smooth crossing and constant-speed orientation climbing tasks; orientation and constant speed means being able to move with the desired forward velocity and desired heading angle, and smooth crossing means that the robot can achieve continuous movement between walls with sudden inclination while maintaining stability. In step S2, by analyzing the constant-speed orientation climbing and wall-crossing tasks, the state space, action space, and reward function for the smooth crossing and constant-speed orientation climbing tasks are defined; the specific steps are as follows: Considering the task requirements and robot characteristics, a 15-dimensional column vector is selected as the state space S, i.e. Where T is the matrix transpose symbol; v x These represent the expected forward velocity and the actual forward velocity of the underwater hexapod robot, respectively; ψ d ψ and ψ represent the expected heading angle and the actual heading angle of the underwater hexapod robot, respectively. θ represents the inclination angle of the wall to be climbed and the pitch angle of the underwater hexapod robot, respectively. The roll angle of the underwater hexapod robot; the motion set is defined as follows: in Let be the joint angular velocity of the i-th blade leg; To enable the underwater hexapod robot to climb at a constant speed and orientation, and to smoothly traverse walls with different inclination angles, the reward function is set as follows: Where λ1, λ2, ..., λ8 are parameters to be adjusted that are greater than zero, t represents the current time, and s t Let a be the state variable at the current moment. t Let g(t) be the action being performed at the current moment, g(t) be the ground-touching stability representation function of the underwater hexapod robot, c1(t) be the switch signal for whether to start crossing the wall, c2(t) be the switch signal for whether to climb the wall, and u(t) be the blade leg utilization representation function. The first four terms of the reward function are set based on the error of the state variables in the state space to achieve constant velocity orientation and posture stability. The last four terms are functions of state variables and motion variables, used to guide the robot to learn stable crossing, and are specifically defined as follows: To ensure that the supporting area formed by the supporting legs is greater than zero, the underwater hexapod robot must have at least three supporting legs, and both sides of the robot's body must have supporting legs. Therefore, the ground contact stability characterization function g is defined as: in, To enable the underwater hexapod robot to smoothly cross walls with different inclination angles, it needs to be able to approach the wall to be climbed and climb the wall at another inclination angle. Therefore, switch signals c1(t) and c2(t) are set in the reward function, as follows: Where D is the distance threshold and Θ is the pitch angle threshold, the values ​​of which are adjusted according to the training effect; Furthermore, to ensure uniform actuation of the six blade legs and prevent individual legs from remaining stationary for extended periods, the range constraint of the blade leg joint angular velocity at each moment must be considered to improve blade leg utilization. The blade leg utilization function u(t) is defined as follows: in These represent the maximum and minimum angular velocities of the six blade leg joints at time t, respectively, and Q represents the angular velocity range threshold, the value of which is adjusted according to the training situation. S3: Using a deep deterministic policy gradient reinforcement learning method, train an underwater hexapod robot to climb at a constant speed and orientation along walls of different inclination angles and to cross smoothly between walls of different inclination angles. S4: In order to improve the stability of the underwater hexapod's movement, on the one hand, a basic thrust is applied to the underwater hexapod's thrusters to ensure that the blade legs do not slip during movement. On the other hand, in order to reduce the influence of restoring force and restoring torque on the underwater hexapod's movement, an additional thrust is applied to the underwater hexapod's thrusters based on the underwater hexapod's current attitude angle to overcome the restoring force and restoring torque.

2. The method for continuous motion control of an underwater hexapod robot on multiple walls based on reinforcement learning according to claim 1, characterized in that: In step S1, for a wall with a known inclination angle, a ranging device fixed to the center of mass of the underwater hexapod robot is used to obtain the forward distance d from the center of mass of the underwater hexapod robot to the wall to be climbed. Simultaneously, a Doppler velocimeter fixed to the center of mass of the underwater hexapod robot is used to obtain the height h of the center of mass of the underwater hexapod robot relative to the wall currently in which the robot is located. Whether the blade leg touches the ground is determined by the height h and the joint angle q of each blade leg of the underwater hexapod robot. i The judgment is as follows: Among them, a i Indicates the ground contact state of the i-th blade leg, a i =1 indicates that the i-th blade leg touches the ground, a i =0 indicates that the i-th blade leg is not touching the ground, where i is the number of the blade leg, i = 1, 2, ..., 6; l r The radius of the blade leg; θ represents the inclination angle of the wall where the i-th blade leg of the underwater robot is located and the pitch angle of the underwater hexapod robot, respectively. b x pi and b y pi These represent the coordinate components of the contact point of the underwater hexapod robot along the x and y directions in the volume coordinate system.

3. The method for continuous motion control of an underwater hexapod robot on multiple walls based on reinforcement learning according to claim 1, characterized in that: In S3, a reinforcement learning method based on deep deterministic policy gradient is used to train a gait that enables the underwater hexapod robot to climb at a constant speed and orientation along walls with different inclination angles and to cross smoothly between walls with different inclination angles. The DDPG algorithm is based on an actor-critic framework, comprising an actor neural network and a critic neural network. Each neural network includes an evaluation network and a target network. The term "heteropolicy" refers to the fact that the evaluation policy actually implemented during training is not the same as the current optimal target policy. This avoids using network parameters for gradient calculation while they are frequently updated, thus improving the stability of the training process and making it easier to converge. The target policy and its corresponding optimal value function are generated by the target network, whose parameters are obtained through soft updates of the evaluation network parameters. Therefore, the target network parameters change relatively little, and the gradient is calculated using the target network. The evaluation policy is generated by the evaluation network, and noise is added to transform it into a stochastic policy, which interacts with the environment and updates the evaluation network parameters using the gradient calculated by the target network. Each interaction accumulates new experience, ensuring sufficient exploration.

4. The method for continuous motion control of an underwater hexapod robot on multiple walls based on reinforcement learning according to claim 1, characterized in that: In S4, in order to improve the stability of the underwater hexapod robot's movement, on the one hand, a basic thrust is applied to the underwater hexapod robot's thrusters to ensure that the blade legs of the underwater hexapod robot do not slip during movement; on the other hand, in order to reduce the influence of restoring force and restoring torque on the underwater hexapod robot's movement, an additional thrust is applied to the underwater hexapod robot's thrusters based on the underwater hexapod robot's current attitude angle to overcome the restoring force and restoring torque. Based on the blade leg joint torque τ of the generated gait i We can obtain: Among them, F τix F τiy These are the components of the force exerted by the blade leg joint torque on the contact point along the x and y directions in the contact point coordinate system, respectively. r The radius of the blade leg; The vertical force F required at the contact point of the blade leg is determined based on the non-slip constraint. Tiy The specific calculation formula is as follows: Where μ is the ground friction coefficient; N i The ground support force at the point of contact; Based on the DH coordinate transformation, the vertical force F required at the contact point of the blade leg is... Tiy The force required to transfer to the robot's center of mass is provided by the thrusters. The components of the thruster thrust along the three directions of the body coordinate system are denoted as T. x T y T z Then F Tiy and T x T y T z The following relationship must be satisfied: Therefore, the basic thrust of the thruster for: in, C represents the base thrust value of the i-th thruster. t Assign a matrix to the thrust; The restoring force and restoring torque of an underwater hexapod robot are determined by gravity G, buoyancy B, and attitude angle. The components of the restoring force in the x, y, and z directions are denoted as F, respectively. gx F gy F gz The components of the restoring torque in the x, y, and z directions are denoted as M, respectively. gx M gy M gz The six components are specifically represented as follows: Where, x c ,y c ,z c Let be the coordinates of the underwater robot's center of buoyancy in the volume coordinate system; Therefore, the additional thrust of the thruster for: in, C represents the additional thrust of the i-th thruster; t Assign a matrix to the thrust; In summary, the total thrust of the i-th thruster is

Images