Real-time dynamic obstacle avoidance method for underactuated AUV in three-dimensional environment

Abstract

The application belongs to the field of underwater robot obstacle avoidance, and particularly relates to a real-time dynamic obstacle avoidance method for underactuated AUV in three-dimensional environment, which comprises the following steps: constructing a maneuverability model of the AUV, calculating the relative position and posture between the AUV and the obstacle; establishing an obstacle risk assessment model, and judging the risk grades of different obstacles to screen out the obstacle with the greatest threat to the AUV; based on the deep deterministic policy gradient algorithm, building a network architecture of the dynamic obstacle avoidance system; designing the input and output of the dynamic obstacle avoidance system to realize the mapping from state to action; inputting the obtained action to the maneuverability model of the AUV to realize the movement of the AUV; setting a reinforcement learning reward function of the dynamic obstacle avoidance system; according to the actual scene, building a virtual simulation environment to train the dynamic obstacle avoidance system, saving the trained model, and through the dynamic obstacle avoidance system, realizing the real-time dynamic obstacle avoidance of the underactuated AUV in the real marine environment.

Description

Technical Field

[0001] This invention belongs to the field of obstacle avoidance for underwater robots, specifically a real-time dynamic obstacle avoidance method in a three-dimensional environment for underactuated AUVs. Background Technology

[0002] The ocean is rich in resources, which is why countries around the world have increased their efforts to develop and utilize it in recent years. However, due to the complex, changeable, and harsh marine environment, most land-based equipment cannot operate underwater. Autonomous underwater vehicles (AUVs), with their high mobility, autonomy, and safety, are widely used in underwater missions.

[0003] In underwater environments, obstacle avoidance by AUVs depends not only on the AUV's motion characteristics, i.e., kinematic and dynamic constraints, but also on environmental factors. In addition to known terrain and static obstacles, due to the complexity and unpredictability of the underwater environment, AUVs cannot obtain complete environmental information before a mission and may encounter unknown obstacles during navigation. Among these obstacles, dynamic obstacles are particularly dangerous when they collide with AUVs because their motion is difficult to predict and they have high kinetic energy, seriously threatening the safety of AUVs.

[0004] Based on the above requirements, AUVs must possess real-time obstacle avoidance capabilities to ensure their own safety and improve mission efficiency. Therefore, a real-time dynamic obstacle avoidance system is needed to enable AUVs to cope with complex and ever-changing underwater environments, avoid static and dynamic obstacles encountered during navigation, and ensure the successful completion of missions.

[0005] Most existing methods are only applied to two-dimensional planes, and some only decouple three-dimensional space. In practice, they are still treated as two-dimensional planes and do not truly consider obstacle avoidance in three-dimensional environments, which affects obstacle avoidance efficiency. At the same time, the obstacle scenarios considered by existing methods are relatively simple, mostly static obstacles, and the relevant information of the obstacles has been obtained in advance, resulting in limited obstacle avoidance capabilities. Some methods consider dynamic obstacles, but only set the obstacles to simple motion states. The trained model is random and lacks universality. In addition, when there are multiple obstacles near the AUV, the search ability of the model is low due to the large state dimension, making it difficult for the system to formulate the optimal strategy. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time dynamic obstacle avoidance method for underactuated underwater robots in a three-dimensional environment. This method enables AUVs to avoid static and dynamic obstacles in real time during operation in a three-dimensional underwater environment. When multiple obstacles exist near the AUV, the risk level of each obstacle is calculated through a risk assessment model, thereby filtering out the obstacles that pose the greatest threat to the AUV and greatly improving obstacle avoidance capabilities. A deep reinforcement learning algorithm is used to construct an end-to-end model from state input to action output. Based on the information obtained by the sensors, the AUV can make decisions and achieve autonomous obstacle avoidance, thus overcoming the shortcomings of existing underwater robots.

[0007] The technical solution adopted by the present invention to achieve the above objectives is: a real-time dynamic obstacle avoidance method in an underactuated AUV three-dimensional environment, comprising the following steps:

[0008] S1: Construct an AUV maneuverability model;

[0009] S2: Calculate the relative position and attitude between the AUV and the obstacle using obstacle information detected by sonar;

[0010] S3: Based on the relative position and attitude between the AUV and the obstacle obtained in step S2, establish an obstacle risk assessment model, determine the risk level of different obstacles, and screen out the obstacles that pose the greatest threat to the AUV.

[0011] S4: Based on the deep deterministic policy gradient algorithm, build the network architecture of the dynamic obstacle avoidance system;

[0012] S5: Design the input and output of the dynamic obstacle avoidance system, transmit the state information to the network architecture of the dynamic obstacle avoidance system, obtain the action output of the system, construct an end-to-end dynamic obstacle avoidance system from state input to action output, realize the mapping from state to action; input the obtained action to the AUV maneuverability model constructed in step S1, thereby realizing the movement of the AUV.

[0013] S6: Set the reinforcement learning reward function for the dynamic obstacle avoidance system;

[0014] S7: Build a virtual simulation environment based on the actual scenario, train the dynamic obstacle avoidance system, save the trained model, and realize real-time dynamic obstacle avoidance of underactuated AUVs in the real marine environment by equipping the dynamic obstacle avoidance system.

[0015] Step S1 specifically includes:

[0016] Constructing a maneuverability model for the AUV, neglecting roll, the AUV's motion in three-dimensional space consists of five parts: advance / retreat, lateral movement, heave, pitch, and yaw. The AUV's position and attitude are then:

[0017] η = [x,y,z,θ,ψ] T

[0018] Where (x,y,z) represent the position coordinates in the geodetic coordinate system, and θ and ψ represent the dip angle and heading angle, respectively;

[0019] The velocity of an AUV is given by v = [u, v, w, q, r] T express;

[0020] Where (u,v,w) represent the longitudinal velocity, lateral velocity and vertical velocity of the AUV, respectively, and q and r represent the pitch angular velocity and the turning angular velocity, respectively.

[0021] The nonlinear kinematics and dynamics equations of an AUV are expressed as follows:

[0022]

[0023]

[0024] Where J(η) is the five-degree-of-freedom coordinate transformation matrix for AUV spatial motion; M represents the inertia matrix, C(v) is the Coriolis centripetal force matrix, D(v) represents the damping matrix, g(η) is the restoring force matrix, and τ represents the control force and torque. For underactuated AUVs, matrix τ is expressed as:

[0025] τ=[τ u ,0,0,τ q ,τ r ] T

[0026] Where τ u τ q τ r These represent the longitudinal thrust, pitch moment, and yaw moment of the AUV, respectively.

[0027] Step S2 includes the following steps:

[0028] 2-1) Obtain the location information η of the obstacle in the geodetic coordinate system using the forward-looking sonar and side-scan sonar mounted on the AUV. o =(x o ,y o ,z o ) T ;

[0029] 2-2) Based on the position coordinates of the AUV and the obstacle, calculate the relative positional relationship between the AUV and the obstacle, i.e.:

[0030] p ao =(x ao ,y ao ,z ao) T =(x o ,y o ,z o ) T -(x,y,z) T

[0031] Where, p ao This represents the position vector from the AUV to the obstacle, (x ao ,y ao ,z ao () represents the coordinates of a vector;

[0032] 2-3) Based on the position vector p of the AUV pointing towards the obstacle ao To obtain the distance between the AUV and the obstacle, i.e., p ao Modulus length:

[0033]

[0034] 2-4) Based on the position vector p of the AUV pointing towards the obstacle ao The relative position vector p is obtained through the coordinate transformation matrix R(θ,ψ). ao Projection p on the AUV's own coordinate system r for:

[0035]

[0036] 2-5) According to projection p r Obtain the relative pose between the AUV and the obstacle, including the relative pitch angle θ. r With a relative heading angle of ψ r ,Right now:

[0037]

[0038]

[0039] 2-6) Based on the position coordinates (x) of the target point g ,y g ,z g Based on steps 2-1) to 2-5), obtain the distance d between the AUV and the target point. rg Relative pitch angle θ rg With relative heading angle ψ rg .

[0040] Step S3 specifically includes:

[0041] 3-1) By determining whether the coordinates of the obstacle change, static obstacles are classified as dynamic obstacles, using O... i Let i represent the characteristics of obstacle i, that is:

[0042]

[0043] 3-2) Construct a risk assessment model for obstacles to determine their risk level; based on the distance d between the AUV and the target point. rg Relative pitch angle θ rg With relative heading angle ψ rg Then the risk level M of the obstacle labeled i i Represented as:

[0044]

[0045] Where k1, k2, k3, m1, m2, and m3 are all proportionality coefficients greater than zero. This indicates the safe threshold for the distance between the AUV and the surface of the obstacle. This represents the distance between the AUV and obstacle i. The radius of obstacle i is represented by f(), which is a user-defined function.

[0046] Simplifying the above equation, we get:

[0047] M i =M com +M con

[0048] Among them, M com Defined as a regular term, M con Defined as a condition item;

[0049] 3-3) When sonar detects multiple obstacles near the AUV, the risk level of each obstacle is obtained, and the risk level M of each obstacle is compared. i Filter out M i The biggest obstacle, namely:

[0050]

[0051] Among them, i max The corresponding obstacles are those that pose the greatest threat to AUVs at present.

[0052] The risk levels of different obstacles are determined as follows:

[0053] a. When the AUV is able to detect obstacles, M com There is always a value, M com Distance d between AUV and obstacle ao Relative pitch angle θ r The relative heading angle is ψ rThe risk level is negatively correlated with the distance between the AUV and the obstacle, the relative pitch angle, and the relative heading angle. The risk level is highest when the following conditions are met.

[0054]

[0055] b. In the obstacle risk assessment model, the condition term M con , is represented as:

[0056]

[0057] Condition item M con In China, k2O i The value ∈[0,k2] is related to the dynamic and static characteristics of obstacles. Dynamic obstacles are considered to be more threatening than static obstacles, thus increasing the risk level of dynamic obstacles.

[0058] When the obstacle is a static obstacle, k2O i =0;

[0059] When the obstacle is a dynamic obstacle, k2O i The value is k2;

[0060] c. Condition Item M con In In this context, its user-defined function f(t) is expressed as:

[0061]

[0062] when hour, Then the value of this item is

[0063] when hour, The value of this item is:

[0064] The decision is made when the distance between the AUV and the obstacle exceeds the set safety threshold and the radius of the obstacle itself. When and when, This is a constant and does not affect the risk level of each obstacle; as the distance between the AUV and the obstacle decreases, when it is less than the sum of the threshold and the obstacle's radius, The value increases as the distance between them decreases;

[0065] when At this time, M i →+∞, treating the AUV as a point mass and the obstacle as a sphere, with the distance between the AUV and the object's surface being the radius of the obstacle itself, indicates that the AUV has collided with the obstacle, at which point the risk level is highest.

[0066] The deep deterministic strategy gradient algorithm is an algorithm based on the AC framework;

[0067] The AC framework consists of a policy network (Actor) and a value network (Critic); each of the policy network (Actor) and the value network (Critic) has an estimation network and a target network, respectively.

[0068] Step S4 specifically includes:

[0069] 4-1) The input to the policy network Actor is the state of the AUV, and the output is the action;

[0070] The estimation network in the policy network Actor is denoted as: a t =μ θ (s t ), where μ θ (s t ) represents strategy, s t With a t These represent the current state and action, respectively, and θ represents the parameters of the estimation network in the Actor.

[0071] For the target network in the policy network Actor, the input is the state s of the next time step. t+1 The output is the action 'a' at the next moment. t+1 , is represented as: in, Represents the target network parameters;

[0072] OU noise is introduced into the estimation network of the policy network (Actor) to increase the randomness of actions, i.e.:

[0073] a t =μ θ (s t )+N t

[0074] Where, N t Represents OU noise;

[0075] 4-2) The actions performed by the AUV are evaluated by fitting the value function Q(s,a) through the value network Critic; where the input of the value estimation network is s. t With a t The output is Q. w (s t ,a t The input to the value target network is s. t+1 a t+1 Output

[0076] 4-3) A memory bank is set up in the network structure. During each training session, the interaction data between the AUV and the environment (s) are stored. t ,a t ,r t ,s t+1 The parameters θ and w are stored in the memory bank. During the update process, N samples are randomly drawn from the memory bank to update the network parameters θ and w.

[0077] 4-4) Value function Q based on Critic fitting of value network w (s t ,a t To update the policy network, we use gradient ascent to maximize the Q-value of the Critic output of the value network. The gradient is then expressed as:

[0078]

[0079] in, Represents the gradient symbol;

[0080] 4-5) By calculating the temporal difference of the value network Critic, and then calculating the mean squared error, gradient descent is used to minimize the objective function and update the network parameters w. The minimized objective function L(w) is expressed as:

[0081]

[0082] Where γ∈(0,1) represents the discount factor;

[0083] 4-6) Update the target network parameters in the policy network (Actor) and value network (Critic) using soft updates. and Right now:

[0084]

[0085]

[0086] Where ξ∈(0,1) represents the moving average coefficient, which can affect and The extent of the update;

[0087] 4-7) Complete the establishment of the policy network (Actor) and value network (Critic) framework in the AUV dynamic obstacle avoidance system.

[0088] The design defines the inputs and outputs of the dynamic obstacle avoidance system, and constructs an end-to-end model from state input to action output to achieve the mapping between the two. Specifically:

[0089] 5-1) The input to the dynamic obstacle avoidance system includes the AUV's own position and attitude information η=[x,y,z,θ,ψ] T And the speed information of the AUV, v = [u, v, w, q, r] T ;

[0090] 5-2) Based on the obstacle risk assessment model, after identifying the most threatening obstacle i, the obstacle's dynamic and static characteristics O are determined. i Distance between AUV and relative pitch angle Relative heading angle obstacle radius The obstacle has a risk level of M. i All data is input into the dynamic obstacle avoidance system; simultaneously, the distance d between the AUV and the target point is obtained. rg Relative pitch angle θ rg With relative heading angle ψ rg Also used as an input to the dynamic obstacle avoidance system, the total input to the dynamic obstacle avoidance system is:

[0091]

[0092] 5-3) s t After normalizing the data as a whole, the input state s from the previous time step is then stacked and processed. t-1 Input state s at the current time t This information is input into the dynamic obstacle avoidance system as a whole.

[0093] 5-4) Control the action value output by the dynamic obstacle avoidance system between (-1, +1) and perform mathematical transformation to output an action that is compatible with the AUV model;

[0094] 5-5) Based on the dynamic obstacle avoidance system, realize state s t To action a t The mapping, that is:

[0095] a t =f(s) t ) s t →a t

[0096] 5-6) The AUV completes the obstacle avoidance task based on the actions output by the dynamic obstacle avoidance system.

[0097] The step involves setting a reinforcement learning reward function based on the dynamic obstacle avoidance system and the inputs and outputs of the designed dynamic obstacle avoidance system, specifically as follows:

[0098] 6-1) During the execution of a mission, an AUV needs to reach a designated target location. Therefore, a terminal reward is established. When the AUV reaches the designated location, it receives a reward, represented as follows:

[0099] r = + r1 r1 > 0

[0100] When an AUV collides with an obstacle, a penalty is imposed, as shown below:

[0101] r = -r² r² > 0

[0102] 6-2) Based on the distance d between the AUV and the target point ag Relative pitch angle θ ag and relative heading angle ψ ag The reward items are set as follows:

[0103]

[0104] Among them, -w1d ag To reduce the distance between the AUV and the target point, To guide the AUV in adjusting its attitude so that it can navigate toward the target location;

[0105] 6-3) Based on the obstacle risk assessment model, the obstacle avoidance task is decomposed into: reducing the threat posed by obstacles to the AUV, and setting the risk value M. i Perform the transformation:

[0106] in, and These represent the risk levels of the most threatening obstacles at the previous and current moments, respectively, obtained through the obstacle risk assessment model.

[0107] If the highest obstacle risk level at the current moment is lower than the highest obstacle risk level at the previous moment, the AUV is given a positive reward value; otherwise, it is given a negative penalty.

[0108] 6-4) Finally, the reinforcement learning reward function is set as follows:

[0109]

[0110] Step S6 specifically includes:

[0111] (1) Training begins; network initialization is performed.

[0112] (2) Perform loop condition judgment: whether the current number of training rounds is less than the maximum number of rounds. If so, execute step (3); otherwise, end the training and execute step (9).

[0113] (3) Initialize the AUV's position, attitude, speed, output force / torque, target point position, obstacle position, and obstacle motion state;

[0114] (4) According to the dynamic obstacle avoidance system, the AUV is based on the current input state s t Select action a t The environment awards it a reward value r, and then it enters the next state s. t+1 ;

[0115] (5) Set the next state s t+1 Assign the value to the current state s t Prepare for the next cycle;

[0116] (6) Determine whether the network update step of the dynamic obstacle avoidance system has been reached. If yes, execute (7); otherwise, skip and execute step (8) directly.

[0117] (7) Extract samples from the memory bank and update the network of the dynamic obstacle avoidance system;

[0118] (8) Perform a loop judgment to see if the round termination condition is met (reaching the target point, touching the obstacle, or the current step number reaches the maximum number of steps in the round). If so, the round ends and the training of the next round begins, returning to step (2). Otherwise, the next time step begins, returning to step (4).

[0119] (9) After training, save the network model of the trained dynamic obstacle avoidance system, and judge whether the training was successful by checking the task completion status and the trend of the reward value curve.

[0120] If the system has not yet obtained a complete obstacle avoidance strategy, the parameters are readjusted and the training continues for the set number of times. If the system has been able to successfully reach the target point and avoid various static and dynamic obstacles, the AUV can dynamically avoid obstacles in real-time in the real marine environment.

[0121] The present invention has the following beneficial effects and advantages:

[0122] 1. This invention provides a real-time dynamic obstacle avoidance method for underactuated underwater robots in a three-dimensional environment. By constructing a maneuverability model of an underactuated AUV, the distance, relative pitch angle, and relative heading angle between the AUV and the obstacle are calculated, thereby obtaining the relative position and attitude information between the AUV and the obstacle. The proposed method can be directly applied to a three-dimensional environment without decoupling to a two-dimensional plane.

[0123] 2. This invention provides an obstacle risk assessment method. When there are multiple obstacles near an AUV, the risk level of each obstacle is calculated through a risk assessment model, thereby filtering out the obstacles that pose the greatest threat to the AUV, which greatly improves obstacle avoidance capability and efficiency. Compared with inputting all obstacle information into the obstacle avoidance system, this method reduces the dimensionality of the input data.

[0124] 3. The dynamic obstacle avoidance system based on the deep deterministic policy gradient algorithm proposed in this invention only requires the state information to be input into the system, and it can output the corresponding decision action end-to-end. Compared with traditional analysis methods, it has stronger search capabilities and higher real-time performance.

[0125] 4. By randomizing the initial environment and setting dynamic obstacles with different motion states, this invention can enhance the system's ability to cope with complex environments and improve model transferability. Attached Figure Description

[0126] Figure 1 This is a flowchart of the method of the present invention;

[0127] Figure 2 This is a schematic diagram of the AUV motion coordinate system of the present invention;

[0128] Figure 3 This is a schematic diagram showing the relative position between the AUV of the present invention and an obstacle;

[0129] Figure 4 This is a network structure framework diagram of the dynamic obstacle avoidance system based on the DDPG algorithm of this invention;

[0130] Figure 5 This is a flowchart of the AUV training method of the present invention. Detailed Implementation

[0131] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0132] like Figure 1 The diagram shown is a flowchart of the method of the present invention. The present invention provides a real-time dynamic obstacle avoidance method for underactuated AUVs in a three-dimensional environment, comprising the following steps:

[0133] S1: Construct an AUV maneuverability model;

[0134] S2: Based on obstacle information detected by sonar, calculate the relative position and attitude between the AUV and the obstacle, which is used for obstacle risk level judgment in step S3;

[0135] S3: Based on the relative position and attitude between the AUV and the obstacle obtained in step S2, establish an obstacle risk assessment model, determine the risk level of different obstacles, and screen out the obstacles that pose the greatest threat to the AUV.

[0136] S4: Based on the deep deterministic policy gradient algorithm, build the network architecture of the dynamic obstacle avoidance system;

[0137] S5: Design the input and output of the dynamic obstacle avoidance system, transmit the state information to the network architecture of the dynamic obstacle avoidance system, obtain the action output of the system, construct an end-to-end dynamic obstacle avoidance system from state input to action output, realize the mapping from state to action; input the obtained action to the AUV maneuverability model constructed in step S1, thereby realizing the movement of the AUV.

[0138] S6: Set the reinforcement learning reward function for the dynamic obstacle avoidance system;

[0139] S7: Build a virtual simulation environment based on the actual scenario, train the dynamic obstacle avoidance system, save the trained model, and realize real-time dynamic obstacle avoidance of underactuated AUVs in the real marine environment by equipping the dynamic obstacle avoidance system.

[0140] (1) In step S1, the construction of the operational model of the underwater robot specifically includes the following steps:

[0141] First, a maneuverability model of the AUV is constructed, ignoring roll. The AUV's motion in three-dimensional space consists of five parts: advance / retreat, lateral movement, heave, pitch, and yaw. The motion reference coordinate system is as follows: Figure 2 As shown, the position and orientation of the AUV are determined by η = [x, y, z, θ, ψ]. T The expression represents the position coordinates in the geodetic coordinate system, where (x, y, z) represent the pitch angle and heading angle, respectively; the velocity is given by v = [u, v, w, q, r]. T Let (u, v, w) represent the longitudinal, lateral, and vertical velocities of the AUV, respectively, and q and r represent the pitch angular velocity and bow angular velocity, respectively. The nonlinear kinematics and dynamic equations of the AUV can be expressed as:

[0142]

[0143]

[0144] Where J(η) is the five-degree-of-freedom coordinate transformation matrix for AUV spatial motion; M represents the inertia matrix, which is derived from the rigid body's own inertia matrix M0.RB With the additional mass matrix M A Composition; C(v) is the Coriolis centripetal force matrix, which is composed of the rigid body Coriolis centripetal force matrix C RB The Coriolis centripetal force matrix C caused by the added mass A Composition; D(v) represents the damping matrix, consisting of linear damping D and nonlinear quadratic damping D. n Composition; g(η) is the restoring force matrix, generated by gravity and buoyancy; τ represents the control force and torque;

[0145] J(η) is represented as:

[0146]

[0147] To simplify the model and facilitate analysis and control, the following assumptions are made:

[0148] An AUV possesses good symmetry, being symmetrical about its three cross-sections, with its center of gravity and center of buoyancy coinciding. Furthermore, the AUV exhibits neutral buoyancy, meaning that gravity W equals buoyancy B; therefore, the product of inertia I... xy =I yx =I xz =0, barycentric coordinates (x g ,y g ,z g )=(0,0,0), and the effect of the restoring force matrix g(η) is negligible;

[0149] Based on the above assumptions, the rigid body's own inertia matrix is ​​represented as M. RB =diag(m,m,m,I) y ,I z ), additional mass matrix

[0150] Where I y I z X is the moment of inertia of the AUV rotating about the y and z axes. . Y . Z . M . N . This is the hydrodynamic coefficient, which is related to factors such as the shape of the AUV itself;

[0151] The Coriolis centripetal force matrix can be expressed as:

[0152]

[0153]

[0154] Where m is the mass of the AUV itself;

[0155] The damping matrix is ​​expressed as:

[0156]

[0157]

[0158] In this invention, the AUV is a combined rudder and propeller type, which has the advantages of long range, easy control at high speed, and low cost, and has wider applicability and fewer hardware requirements.

[0159] Therefore, for underactuated AUVs, the matrix τ can be represented as:

[0160] τ=[τ u ,0,0,τ q ,τ r ] T (8)

[0161] Where τ u τ q τ r These represent longitudinal thrust, pitching moment, and yaw moment, respectively.

[0162] (2) In step S2, based on the obstacle information detected by sonar, the relative position and attitude between the AUV and the obstacle are calculated for use in step S3 to determine the obstacle risk level. Specifically, this includes the following steps:

[0163] Figure 3 This diagram illustrates the relative position between the AUV and the obstacle. Based on forward-looking sonar and side-scan sonar, the position information η of the obstacle in the geodetic coordinate system is obtained. o =(x o ,y o ,z o ) T ;

[0164] Based on the position coordinates of the AUV and the obstacle, calculate the relative positional relationship between the AUV and the obstacle:

[0165] p ao =(x ao ,y ao ,z ao ) T =(x o ,y o ,z o ) T -(x,y,z) T

[0166] Where, p a o represents the position vector pointing from the AUV to the obstacle, (x ao ,y ao ,z ao () represents the coordinates of a vector;

[0167] Based on vector p ao The distance between the AUV and the obstacle, i.e., p, can be calculated. ao The modulus is expressed as:

[0168]

[0169] Based on vector p ao The relative position vector p is obtained through the coordinate transformation matrix R(θ,ψ). ao The projection p on the rigid body's own coordinate system r for:

[0170]

[0171] Furthermore, based on p r The relative pitch angle θ is obtained. r With a relative heading angle of ψ r for:

[0172]

[0173]

[0174] Similarly, let the position coordinates of the target point be (x... g ,y g ,z g Based on the above settings, the distance d between the AUV and the target point is calculated. rg Relative pitch angle θ rg With relative heading angle ψ rg ;

[0175] (3) In step S3, an obstacle risk assessment model is established. Based on factors such as the relative distance, relative pitch angle, and relative heading angle between the AUV and the obstacle, the risk level of different obstacles is determined, and the obstacle that poses the greatest threat to the AUV is selected.

[0176] At the same time, sonar may detect multiple obstacles. Inputting all their information into the obstacle avoidance system would be too complex and require too much data. Therefore, it is necessary to establish an obstacle risk assessment model to screen out the most threatening obstacles.

[0177] Based on sonar image algorithms and location estimation principles, static and dynamic obstacles are distinguished by determining whether their coordinates change. (O) i The characteristics of obstacle i are represented as follows:

[0178]

[0179] Furthermore, a risk assessment model for obstacles is constructed to determine the risk level of obstacles. Based on the relative distance, relative pitch angle, and relative heading angle between the AUV and the obstacle calculated in step 2, the risk level M of obstacle i is determined. i Represented as:

[0180]

[0181] Where k1, k2, k3, m1, m2, m3 > 0 represent proportionality coefficients. This represents the safe threshold for the distance between the AUV and the surface of the obstacle, where f() is a user-defined function;

[0182] Regarding the obstacle risk assessment model, the following section will introduce each part of the expression in turn:

[0183] First item:

[0184]

[0185] This factor is related to the relative position and attitude between the AUV and the obstacle, and its value decreases negatively with increasing distance between the AUV and the obstacle, relative pitch angle, and relative heading angle. The risk level is highest when the following conditions are met.

[0186]

[0187] When the distance between the AUV and the obstacle, the relative pitch angle, and the relative heading angle are large, the AUV is relatively safe, and these values ​​are small. When the AUV can detect the obstacle, M... com The first term always has a value and is therefore called the regular term. However, in the obstacle risk assessment model, the second and third terms are only triggered under certain conditions and are therefore collectively referred to as the conditional terms M. con , is represented as:

[0188]

[0189] Among them, the second term k2O i ∈[0,k2] is related to the dynamic and static characteristics of the obstacle. When it is a static obstacle, this term is 0, and when it is a dynamic obstacle, the value of this term is k2. Because it is more difficult to predict the motion state of dynamic obstacles in the next moment, and the damage is greater when a collision occurs, dynamic obstacles are generally considered to be more threatening than static obstacles, thus increasing the risk level of dynamic obstacles.

[0190] Furthermore, the third item As a condition term M conPart of the measure is to increase the risk level when the distance between an AUV and an obstacle surface is less than a safety threshold; because the safety threshold represents the minimum safe distance between the AUV and the obstacle surface, and when the distance is less than the threshold... At this point, it indicates that the AUV's current situation is extremely dangerous, with a very high probability of colliding with obstacles, while the standard item M... com This cannot meet the requirement because The change is linear, it is consistent with... Together they influence the magnitude of the first term, and When the value fluctuates around the threshold, it does not cause the obstacle hazard level M to be triggered. i The mutation therefore led to the addition of a third term, which increases the danger level when the distance is less than the safety threshold;

[0191] The user-defined function f(t) is expressed as:

[0192]

[0193] Therefore when hour, The value of the third item is when hour, The value of this item is Analysis shows that when the distance between the AUV and the obstacle exceeds the sum of the set safety threshold and the obstacle's own radius, this term remains constant and does not affect the hazard level of each obstacle; however, as the distance decreases and falls below the sum of the threshold and the obstacle's radius, the value of the third term changes, increasing as the distance between them decreases; when... At this time, M i →+∞, treating the AUV as a point mass and the obstacle as a sphere, the distance between the AUV and the object's surface is the radius of the obstacle itself, indicating that the AUV has collided with the obstacle, at which point the risk level is the highest;

[0194] The above analysis explains the meaning of each item in the risk assessment model. Furthermore, when the sensor detects multiple obstacles near the AUV, the threat level M of each obstacle is compared. i Filter out the largest one:

[0195]

[0196] i max The corresponding obstacles are those that pose the greatest threat to AUVs at present.

[0197] (4) In step S4, an underwater robot dynamic obstacle avoidance system is built based on the depth deterministic strategy gradient algorithm.

[0198] The specific steps are as follows:

[0199] Deep deterministic policy gradient (DDPG) is an algorithm based on the actor-critic (AC) framework, comprising actor and critic networks, i.e., the policy network and the value network, respectively. Figure 4 As shown, there is an estimation network and a target network in Actor and Critic respectively, so the algorithm framework contains a total of 4 neural networks;

[0200] The policy network takes the agent's state as input and outputs actions; the estimation network is denoted as a. t =μ θ (s t ), where μ θ (s t ) represents strategy, s t With a t These represent the current state and action, respectively, and θ represents the parameters of the estimation network in the Actor.

[0201] For the target network in the Actor, the input is the state s at the next time step. t+1 The output is the action 'a' at the next moment. t+1 , is represented as: in, Represents the target network parameters;

[0202] Because the DDPG algorithm is a deterministic strategy algorithm with poor randomness, Ornstein-Uhlenbeck (OU) noise is introduced to increase the randomness of the actions, expressed as:

[0203] a t =μ θ (s t )+N t (19)

[0204] Where, N t Represents OU noise;

[0205] Furthermore, the actions performed by the agent are evaluated by fitting a value function Q(s,a) to a value network; the input to the value estimation network is s. t With a t The output is Q. w (s t ,a t The input to the target network is s. t+1 a t+1 Output

[0206] Furthermore, a memory bank is set up in the network structure, and during each training session, the interaction data between the AUV and the environment (s) are stored. t ,a t ,r t ,s t+1 The network parameters θ and w are stored in the memory. During the update process, N samples are randomly drawn from the memory to update the network parameters θ and w.

[0207] Based on the value function fitted by the value network, the policy network is updated. Gradient ascent is used to maximize the Q-value of the value network output, and the gradient is expressed as:

[0208]

[0209] in, Represents the gradient symbol;

[0210] The value network updates its network parameters w by calculating temporal differences, then calculating the mean squared error, and finally using gradient descent to minimize the objective function. The objective function L(w) is expressed as:

[0211]

[0212] In the formula, γ∈(0,1) represents the discount factor;

[0213] Furthermore, the target network parameters in the Actor and Critic are updated using a soft update method. and

[0214]

[0215]

[0216] Where ξ∈(0,1) represents the moving average coefficient, which can affect and The extent of the update;

[0217] (5) Design the input and output of the dynamic obstacle avoidance system, transmit the state information to the network architecture of the dynamic obstacle avoidance system, obtain the action output of the system, construct an end-to-end dynamic obstacle avoidance system from state input to action output, realize the mapping from state to action; input the obtained action to the AUV maneuverability model constructed in step S1, thereby realizing the movement of the AUV.

[0218] The input to the dynamic obstacle avoidance system should include the AUV's own position and attitude information η = [x, y, z, θ, ψ]. T The speed information of the AUV is v = [u, v, w, q, r]. T ;

[0219] Furthermore, based on step S3, it is known that inputting all obstacle information into the system will result in an excessively large input dimension. At the same time, some obstacles that are far away and pose little threat to the AUV will be used as interference items and will affect the AUV's decision-making after being input into the system.

[0220] Therefore, based on the risk assessment model, after identifying the most threatening obstacle i, the obstacle's dynamic and static characteristics O are... i Distance between AUV and relative pitch angle Relative heading angle obstacle radius The obstacle has a risk level of M. i All data is input into the dynamic obstacle avoidance system; simultaneously, the distance d between the AUV and the target point is obtained. rg Relative pitch angle θ rg With relative heading angle ψ rg Also used as an input to the dynamic obstacle avoidance system, the total input to the dynamic obstacle avoidance system is:

[0221]

[0222] s t As a whole, before being input into the dynamic obstacle avoidance system, normalization and model stacking are performed. The purpose of normalization is to improve computational efficiency and prevent gradient explosion. Through model stacking, the input state s from the previous time step is then processed. t-1 Input state s at the current time t This information is input into the dynamic obstacle avoidance system as a whole.

[0223] Furthermore, in this invention, the AUV is an underactuated type. As can be seen from step S1, the system needs to output longitudinal thrust, pitch moment and yaw moment to control the navigation of the AUV.

[0224] In actual operation, it is necessary to impose certain constraints on the action value output by the system. Otherwise, the output action value may have no actual physical meaning or exceed the maximum output range of the actuator. Therefore, the action value output by the system is controlled between (-1, +1) and then mathematically transformed to obtain the ideal action range, thereby outputting an action that is compatible with the AUV model.

[0225] Furthermore, based on the dynamic obstacle avoidance model, state s is realized. t To action a t The mapping, that is:

[0226] a t =f(s) t ) s t →a t (25)

[0227] The AUV completes the obstacle avoidance task based on the actions output by the dynamic obstacle avoidance system.

[0228] (6) In step S6, based on the dynamic obstacle avoidance system and the system's input and output, a reinforcement learning reward function is set, specifically as follows:

[0229] The reward function is closely related to the task requirements; therefore, in this embodiment, the reward function is set as follows:

[0230] During mission execution, AUVs need to reach a designated target location. Therefore, a terminal reward system is established. When an AUV reaches the designated location, it receives a substantial reward, represented as follows:

[0231] r = + r1 r1 > 0 (26)

[0232] Similarly, when an AUV collides with an obstacle, it is given a larger penalty, as shown below:

[0233] r = -r² r² > 0 (27)

[0234] Furthermore, in order to improve the problem of sparse rewards in obstacle avoidance tasks and make the rewards denser, the task objective is decomposed into sub-objectives;

[0235] Reaching the target point is decomposed into a task of approaching the target point, based on the distance d between the AUV and the target point. ag Relative pitch angle θ ag and relative heading angle ψ ag The reward items are set as follows:

[0236]

[0237] The formula consists of three parts, -w1d ag This item aims to reduce the distance between the AUV and the target point. Both methods aim to guide the AUV to adjust its attitude and navigate toward the target location, with the goal of minimizing penalties.

[0238] Furthermore, for obstacle avoidance, the training objective is to enable the AUV to avoid threats posed by obstacles by adjusting its speed and attitude. Therefore, based on the obstacle risk assessment model in step S3, it is not necessary to reset the reward function; it is only necessary to decompose the obstacle avoidance task into the objective of reducing the threat posed by obstacles to the AUV, and assign the risk value M... i Perform the transformation:

[0239]

[0240] In the formula, and Let i and y represent the risk levels of the most threatening obstacles at the previous and current times, respectively, obtained through the obstacle risk assessment model. Note that i at the previous and current times... max It may not be the same obstacle. It must be calculated through a risk assessment model. Just because the obstacle that poses the greatest threat to the AUV at the current moment is i, we cannot only calculate the risk level of obstacle i in the subsequent process. Otherwise, the AUV may fall into greater danger in the decision-making process in order to avoid the current obstacle.

[0241] The above can be understood as follows: at the previous moment, based on the risk assessment model, the obstacle with the highest risk level was m1, and the risk level was... After the AUV performs action a1, the risk level of m1 at the current moment becomes From the results, the AUV's decision was correct because the risk level of obstacle m1 decreased; if the obstacle with the highest risk level at the current moment is m2, the risk level would be... and Although the AUV reduces the threat posed by obstacle m1 through decision-making, it shifts to a greater danger. Therefore, in formula (28), i at time t-1 and time t... max These are the obstacles with the highest risk level at the current decision-making moment;

[0242] If the highest obstacle risk level at the current moment is lower than the highest obstacle risk level at the previous moment, the AUV is given a positive reward value; otherwise, it is given a negative penalty.

[0243] In summary, the reward function can be expressed as:

[0244]

[0245] (7) Step S7: Build a virtual simulation environment based on the actual scene, train the dynamic obstacle avoidance system, save the trained model, and by equipping the dynamic obstacle avoidance system, the underactuated AUV can achieve real-time dynamic obstacle avoidance in the real marine environment.

[0246] A virtual simulation environment is built based on real-world scenarios. By setting the target point generation location, changing the initial position and attitude of the AUV, and setting up static and dynamic obstacles with different motion forms (linear motion, curvilinear motion, uniform motion, uniform acceleration, etc.), the randomness of the environment is enhanced, and the adaptability of the trained system to the environment is improved to cope with complex and ever-changing environments.

[0247] Furthermore, experimental parameters (maximum number of training rounds, maximum number of training steps per round, etc.) and system parameters (network parameters, obstacle risk assessment model coefficients, reward function coefficients, etc.) are set, and then the system is trained.

[0248] like Figure 5 The training process for AUVs can be described as follows:

[0249] (1) Training begins; network initialization is performed.

[0250] (2) Perform loop condition judgment: whether the current number of training rounds is less than the maximum number of rounds. If so, execute step (3); otherwise, end the training and execute step (9).

[0251] (3) Initialize the AUV's position, attitude, speed, output force / torque, target point position, obstacle position, and obstacle motion state;

[0252] (4) According to the dynamic obstacle avoidance system, the AUV is based on the current input state s t Select action a t The environment awards it a reward value r, and then it enters the next state s. t+1 ;

[0253] (5) Set the next state s t+1 Assign the value to the current state s t Prepare for the next cycle;

[0254] (6) Determine whether the network update step of the dynamic obstacle avoidance system has been reached. If yes, execute (7); otherwise, skip and execute step (8) directly.

[0255] (7) Extract samples from the memory bank and update the network of the dynamic obstacle avoidance system;

[0256] (8) Perform a loop judgment to see if the round termination condition is met (reaching the target point, touching the obstacle, or the current step number reaches the maximum number of steps in the round). If so, the round ends and the training of the next round begins, returning to step (2). Otherwise, the next time step begins, returning to step (4).

[0257] (9) After training, save the network model of the trained dynamic obstacle avoidance system, and judge whether the training was successful by checking the task completion status and the trend of the reward value curve.

[0258] If the system has not yet obtained a complete obstacle avoidance strategy, the parameters are readjusted and the training continues for a set number of times. If the system can successfully reach the target point and avoid various static and dynamic obstacles, then the dynamic obstacle avoidance system can cope with the complex and ever-changing marine environment. When the system is moved into the real environment, the robot obtains state information through various sensors and inputs it into the system. The AUV can then obtain the obstacle avoidance strategy and change its course and attitude by adjusting its speed and angular velocity, thereby achieving real-time dynamic obstacle avoidance in the real marine environment.

[0259] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, extensions, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A real-time dynamic obstacle avoidance method for underactuated AUVs in a three-dimensional environment, characterized in that, Includes the following steps: S1: Construct an AUV maneuverability model; S2: Calculate the relative position and attitude between the AUV and the obstacle using obstacle information detected by sonar; S3: Based on the relative position and attitude between the AUV and the obstacle obtained in step S2, establish an obstacle risk assessment model, determine the risk level of different obstacles, and screen out the obstacles that pose the greatest threat to the AUV. Step S3 specifically includes: 3-1) By determining whether the coordinates of the obstacle change, static obstacles are classified as dynamic obstacles. To represent obstacles Its characteristics are: ； 3-2) Construct a risk assessment model for obstacles to determine their risk level; Based on the distance between the AUV and the target point Relative pitch angle relative heading angle The label is Risk level of obstacles Represented as: ； in, , , , , , All are proportionality coefficients that are greater than zero. This indicates the safe threshold for the distance between the AUV and the surface of the obstacle. Represents AUV and obstacles distance, Representing obstacles radius, This is a user-defined function; and This refers to AUV and the first The relative pitch angle and relative heading angle between the obstacles; Simplifying the above formula: ； in, Defined as a regular item, Defined as a condition item; 3-3) When sonar detects multiple obstacles near the AUV, the risk level of each obstacle is obtained, and the risk levels of each obstacle are compared. Filter out The biggest obstacle is: ； in, The corresponding obstacles are those that currently pose the greatest threat to AUVs; S4: Based on the deep deterministic policy gradient algorithm, build the network architecture of the dynamic obstacle avoidance system; S5: Design the input and output of the dynamic obstacle avoidance system, transmit the state information to the network architecture of the dynamic obstacle avoidance system, obtain the action output of the system, construct an end-to-end dynamic obstacle avoidance system from state input to action output, realize the mapping from state to action; input the obtained action to the AUV maneuverability model constructed in step S1, thereby realizing the movement of the AUV. S6: Set the reinforcement learning reward function for the dynamic obstacle avoidance system; S7: Build a virtual simulation environment based on the actual scenario, train the dynamic obstacle avoidance system, save the trained model, and realize real-time dynamic obstacle avoidance of underactuated AUVs in the real marine environment by equipping the dynamic obstacle avoidance system.

2. The real-time dynamic obstacle avoidance method for an underactuated AUV in a three-dimensional environment according to claim 1, characterized in that, Step S1 specifically includes: Constructing a maneuverability model for the AUV, neglecting roll, the AUV's motion in three-dimensional space consists of five parts: advance / retreat, lateral movement, heave, pitch, and yaw. The AUV's position and attitude are then: ； in, Represents the position coordinates in the geodetic coordinate system. and These represent the pitch angle and heading angle, respectively. The speed of AUV is determined by express; in, These represent the longitudinal velocity, lateral velocity, and vertical velocity of the AUV, respectively. and These represent the pitch rate and the turning rate, respectively. The nonlinear kinematics and dynamics equations of an AUV are expressed as follows: ； ； in, It is the five-degree-of-freedom coordinate transformation matrix for AUV spatial motion; Represents the inertia matrix. The centripetal force matrix is ​​represented by the Coriolis formula. Represents the damping matrix. For the restoring force matrix, Represents control force and torque; for underactuated AUVs, the matrix... Represented as: ； in , , These represent the longitudinal thrust, pitch moment, and yaw moment of the AUV, respectively.

3. The real-time dynamic obstacle avoidance method for an underactuated AUV in a three-dimensional environment according to claim 1, characterized in that, Step S2 includes the following steps: 2-1) Obtain the location information of obstacles in the geodetic coordinate system using the forward-looking sonar and side-scan sonar mounted on the AUV. ; 2-2) Based on the AUV's position coordinates and the obstacle's position coordinates, calculate the relative positional relationship between the AUV and the obstacle, i.e.: ； in, This represents the position vector from the AUV to the obstacle. The coordinates represent the vector. Represents the position coordinates in the geodetic coordinate system. and These represent the pitch angle and heading angle, respectively. 2-3) Based on the position vector of the AUV pointing towards the obstacle To obtain the distance between the AUV and the obstacle, i.e. Modulus length: ； 2-4) Based on the position vector of the AUV pointing towards the obstacle Through coordinate transformation matrix To obtain the relative position vector Projection on the AUV's own coordinate system for: ； 2-5) According to the projection Obtain the relative pose between the AUV and the obstacle, including the relative pitch angle. With relative heading angle ,Right now: ； ； 2-6) Based on the position coordinates of the target point Based on steps 2-1) to 2-5), obtain the distance between the AUV and the target point. Relative pitch angle relative heading angle .

4. The real-time dynamic obstacle avoidance method for an underactuated AUV in a three-dimensional environment according to claim 1, characterized in that, The risk levels of different obstacles are determined as follows: a. When the AUV is able to detect obstacles It always has value. Distance between AUV and obstacles Relative pitch angle Relative heading angle is The risk level is negatively correlated with the distance between the AUV and the obstacle, the relative pitch angle, and the relative heading angle. The risk level is highest when the following conditions are met. ； b. In the obstacle risk assessment model, the conditional term , represented as: ； Conditions middle, Related to the dynamic and static characteristics of obstacles, dynamic obstacles are judged to be more threatening than static obstacles, thus increasing the risk level of dynamic obstacles; When the obstacle is a static obstacle, =0; When the obstacle is a dynamic obstacle The value is ; c. Conditional items In In, its user-defined functions Represented as: ； when hour, Then the value of this item is ; when hour, The value of this item is: ; The decision is made when the distance between the AUV and the obstacle exceeds the set safety threshold and the radius of the obstacle itself. When and when, This is a constant and does not affect the risk level of each obstacle; as the distance between the AUV and the obstacle decreases, when it is less than the sum of the threshold and the obstacle's radius, The value increases as the distance between them decreases; when At this time If we consider the AUV as a point mass and the obstacle as a sphere, and the distance between the AUV and the object's surface is equal to the radius of the obstacle, then the AUV has collided with the obstacle, and the risk level is at its highest.

5. The real-time dynamic obstacle avoidance method for an underactuated AUV in a three-dimensional environment according to claim 1, characterized in that, The deep deterministic strategy gradient algorithm is an algorithm based on the AC framework; The AC framework consists of a policy network (Actor) and a value network (Critic); each of the policy network (Actor) and the value network (Critic) has an estimation network and a target network, respectively.

6. The real-time dynamic obstacle avoidance method for an underactuated AUV in a three-dimensional environment according to claim 1, characterized in that, Step S4 specifically includes: 4-1) The input to the policy network Actor is the state of the AUV, and the output is the action; The estimation network in the policy network Actor is represented as: ,in, Representative strategy, and These represent the current state and action, respectively. Represents the parameters of the network estimated in the Actor; For the target network in the policy network Actor, the input is the state at the next time step. The output is the action at the next moment. , represented as: ,in, Represents the target network parameters; OU noise is introduced into the estimation network of the policy network (Actor) to increase the randomness of actions, i.e.: ； in, Represents OU noise; 4-2) Fitting the value function using the value network Critic The actions performed by the AUV are evaluated; the input to the value estimation network is... and The output is The input to the value target network is , Output ; 4-3) A memory bank is set up in the network structure. During each training session, the interaction data between the AUV and the environment is stored. The parameters are stored in a memory bank. During the update process, N samples are randomly drawn from the memory bank to update the network parameters. and ; 4-4) Value function based on Critic fitting of value network The policy network is updated using gradient ascent to improve the output of the value network Critic. To maximize the value, the gradient is expressed as: ； in, Represents the gradient symbol; 4-5) By calculating the temporal difference of the value network Critic, and then calculating the mean squared error, the objective function is minimized using gradient descent to update the network parameters. Where, minimizing the objective function Represented as: ； in, Represents the discount factor; 4-6) Update the target network parameters in the policy network (Actor) and value network (Critic) using soft updates. and ,Right now: ； ； in, This represents the moving average coefficient, which can affect... and The extent of the update; 4-7) Complete the establishment of the policy network (Actor) and value network (Critic) framework in the AUV dynamic obstacle avoidance system.

7. The real-time dynamic obstacle avoidance method for an underactuated AUV in a three-dimensional environment according to claim 1, characterized in that, The design defines the input and output of the dynamic obstacle avoidance system, and constructs an end-to-end model from state input to action output to achieve the mapping between the two. Specifically: 5-1) The input to the dynamic obstacle avoidance system includes the AUV's own position and attitude information. and AUV speed information ; in, Represents the position coordinates in the geodetic coordinate system. and These represent the pitch angle and heading angle, respectively. These represent the longitudinal velocity, lateral velocity, and vertical velocity of the AUV, respectively. and These represent the pitch rate and the turning rate, respectively. 5-2) Based on the obstacle risk assessment model, identify the most threatening obstacles. Then, the obstacle's own dynamic and static characteristics are considered. Distance between AUV and Relative pitch angle Relative bow angle obstacle radius The risk level of the obstacle All data is input into the dynamic obstacle avoidance system; simultaneously, the distance between the AUV and the target point is acquired. Relative pitch angle relative heading angle Also used as an input to the dynamic obstacle avoidance system, the total input to the dynamic obstacle avoidance system is: ； 5-3) After normalizing the data as a whole, the input state from the previous time step is then stacked and processed. Current input state This information is input into the dynamic obstacle avoidance system as a whole. 5-4) Control the action value output by the dynamic obstacle avoidance system within... The mathematical transformations are performed on the data to output actions that are adapted to the AUV model. 5-5) Based on the dynamic obstacle avoidance system, realize the state To action The mapping, that is: ； 5-6) The AUV completes the obstacle avoidance task based on the actions output by the dynamic obstacle avoidance system.

8. The real-time dynamic obstacle avoidance method for an underactuated AUV in a three-dimensional environment according to claim 1, characterized in that, The step involves setting a reinforcement learning reward function based on the dynamic obstacle avoidance system and the inputs and outputs of the designed dynamic obstacle avoidance system, specifically as follows: 6-1) During the execution of a mission, the AUV needs to reach a designated target location. Therefore, a terminal reward is established. When the AUV reaches the designated location, it receives a reward, represented as follows: ； When an AUV collides with an obstacle, a penalty is imposed, as shown below: y ； 6-2) Based on the distance between the AUV and the target point Relative pitch angle and relative heading angle The reward items are set as follows: ； in, To reduce the distance between the AUV and the target point, To guide the AUV in adjusting its attitude so that it can navigate toward the target location; 6-3) Based on the obstacle risk assessment model, the obstacle avoidance task is decomposed into: reducing the threat posed by obstacles to the AUV, and assigning risk values... Perform the transformation: ； Among them, y and These represent the risk levels of the most threatening obstacles at the previous and current moments, respectively, obtained through the obstacle risk assessment model. , , All are proportionality coefficients that are greater than zero; This indicates the safe threshold for the distance between the AUV and the surface of the obstacle; If the highest obstacle risk level at the current moment is lower than the highest obstacle risk level at the previous moment, the AUV is given a positive reward value; otherwise, it is given a negative penalty. 6-4) Finally, the reinforcement learning reward function is set as follows: 。 9. A real-time dynamic obstacle avoidance method in a three-dimensional environment under underactuated AUV according to claim 1, characterized in that, Step S6 specifically includes: (1) Training begins; network is initialized. (2) Perform loop condition judgment: whether the current number of training rounds is less than the maximum number of rounds. If so, execute step (3); otherwise, end the training and execute step (9). (3) Initialize the AUV's position, attitude, velocity, output force / torque, target point position, obstacle position, and obstacle motion state; (4) According to the dynamic obstacle avoidance system, the AUV is based on the current input state. Select Action The environment gives it a reward value Then it enters the next state. ; (5) Change the next state Assign a value to the current state Prepare for the next cycle; (6) Determine whether the network update step of the dynamic obstacle avoidance system has been reached. If yes, execute (7); otherwise, skip and execute step (8) directly. (7) Extract samples from the memory bank and update the network of the dynamic obstacle avoidance system; (8) Perform a loop judgment to determine whether the round termination condition is met (reaching the target point, touching the obstacle, or the current step number reaches the maximum number of steps in the round). If so, the round ends and the next round of training begins, returning to step (2). Otherwise, the next time step begins, returning to step (4). (9) After training, save the network model of the trained dynamic obstacle avoidance system. By checking the task completion status and the trend of the reward value curve, determine whether the training was successful. If the system has not yet obtained a complete obstacle avoidance strategy, then readjust the parameters and continue to perform the training for the set number of times; If the system has successfully reached the target point and avoided various static and dynamic obstacles, the AUV can dynamically avoid obstacles in real-time in the real marine environment.

Images