AUV Hull Navigation Method Based on Deep Reinforcement Learning

Abstract

This invention discloses an AUV hull docking method based on deep reinforcement learning, including: s1: real-time acquisition of the current state s t Using the current parameterized strategy network sampling action a t , will a t The probability density value is denoted as p(a t ); s2: will a t Substitute into the dynamic equation and calculate the state s at the next time step. t+1 and reward function r t s1: Form tuples and store them in the experience pool; s2: If the number of tuples in the experience pool meets the condition, go to s3; otherwise, go to s1; s3: Transfer the tuples in the experience pool to s4. t s t+1 The input is fed into the state-value network to obtain function values, and the advantage function is calculated. b is sampled from the experience pool. s Each tuple is used to perform gradient descent on the parameters of the state-value network using temporal difference error to achieve policy evaluation; s5: Sample b from the experience pool. s Each tuple is used to introduce a rollback mechanism to perform gradient descent on the parameters of the policy network, thereby improving the policy; s6: AUV ends when the termination condition is met, otherwise it goes to s1.

Description

Technical Field

[0001] This invention belongs to the field of underwater vehicle docking technology, and particularly relates to a docking method for AUV hulls based on deep reinforcement learning. Background Technology

[0002] In recent years, the growing enthusiasm for ocean exploration has led to a surge in demand for AUVs (Autonomous Underwater Vehicles). As underwater missions become increasingly challenging, higher requirements are being placed on AUVs' autonomy, endurance, and data storage capabilities. However, AUVs are limited by hardware constraints, which restrict their endurance and data storage capacity. Therefore, refueling and data transfer for AUVs are crucial for long-distance missions. Typically, AUVs need to dock with a mobile or fixed dock for refueling and data transfer. However, the complex dynamics of AUVs make docking a challenging process. Therefore, researching the docking problem of underwater AUVs has significant theoretical and practical implications.

[0003] Patents have researched the docking problem of AUVs. For example, Chinese patent application CN107065882A designs a method for the dynamic autonomous recovery of UUVs (Unmanned Underwater Vehicles) from USVs (unmanned surface vessels). This method divides the docking problem into two processes: first, the USV approaches the UUV in a straight line and the UUV remains stationary; second, when the distance between the two meets certain conditions, the UUV maneuvers to cooperate with the USV for the docking mission. Patent CN112684811A designs a reciprocating docking route control method, which allows the UUV to continuously navigate to the vicinity of the docking device, then moves away from the target according to the docking interface orientation, and turns around after a certain distance, so that the UUV is on the central axis of the docking device, providing a good attitude for end-of-course guidance. Chinese patent application CN112486168A proposes a mobile docking trajectory planning method based on a rotating circle. This method designs different trajectories according to the geometric relationship between the AUV and the docking device. The trajectory is a series of straight and circular trajectories joined together, with the two trajectories tangent at the joining points, ensuring a relatively smooth trajectory and facilitating subsequent path tracking by the AUV. Chinese patent application CN114815818A invents a dynamic path planning method and device for underactuated AUV mobile docking. In the far-end homing phase, it uses a pure tracking method to track pre-set waypoints. In the near-end homing phase, it uses a model predictive control-based method for local planning, solving the optimal problem under constraints, thereby improving docking stability and accuracy in non-ideal underwater environments.

[0004] A review of the patent applications revealed that while the AUV docking problem has been studied in considerable detail, some shortcomings remain. First, some research and inventions rely on manually designed rules for trajectory planning, which diminishes the AUV's autonomy and adaptability, and fails to fully guarantee that the planned trajectory meets certain task requirements (e.g., the shortest or most energy-efficient trajectory). Second, some research and inventions employ simplistic problem modeling. While AUV docking is typically modeled in a three-dimensional environment, some studies model it in a two-dimensional plane or a simplified three-dimensional environment (e.g., ignoring variations in a degree of freedom in three-dimensional space), leading to significant discrepancies between the actual trajectory and the simulation result. Third, some simulations only plan the problem at the kinematic level, neglecting the impact of dynamic constraints. Finally, some traditional model-based control methods suffer from poor practical performance due to the difficulty in obtaining the AUV's various dynamic parameters; while model-free methods, such as reinforcement learning and genetic algorithms, still suffer from slow convergence and a tendency to get trapped in local optima. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an AUV hull docking method based on deep reinforcement learning. This invention first establishes a three-dimensional AUV dynamic model, a dock kinematic model, and an ocean current model; secondly, it designs an action space, state space, and reward function specifically for this problem; thirdly, it improves the parameter update strategy of the policy network based on the PPO algorithm; finally, simulations verify that the improved algorithm outperforms the unimproved algorithm in terms of convergence speed and docking capability.

[0006] In view of this, the present invention proposes a method for docking AUV hulls based on deep reinforcement learning, comprising:

[0007] Step 1: Real-time acquisition of the AUV's current state s based on sonar and its own inertial navigation information. t Utilizing the current parameterized policy network π θ (a t |s t Sampling action a t and a t The probability density value corresponding to the policy distribution is denoted as p(a). t );

[0008] Step 2: Perform action a t Substituting these values ​​into the AUV dynamics equations and combining them with the dock conditions, the state s at the next moment is calculated. t+1 and reward function r t , tuple t ,a​t ,r t ,s t+1 ,p(a t Store in the experience pool;

[0009] Step 3: If the number of tuples in the experience pool meets the parameter update conditions, proceed to step 4; otherwise, go to step 1.

[0010] Step 4: Extract the current state s from the experience pool tuple data. t The state s at the next moment t+1 The input is fed into the state-value network to obtain the state-value function value, which is then used to calculate the advantage function. Sample b from the experience pool s Each tuple is used to perform gradient descent on the parameter φ of the state value network using temporal difference error, thereby achieving policy evaluation;

[0011] Step 5: Sample b from the experience pool again. s Each tuple is used to introduce a rollback mechanism to perform gradient descent on the parameters θ of the policy network, thereby improving the policy.

[0012] Step 6: If the AUV meets the termination condition, the process ends; otherwise, proceed to step 1.

[0013] Preferably, the policy network and the state value network are both independent feedforward neural networks, wherein the input of the policy network is the current state s of the AUV. t The output is the strategy action a. t In state s t The probability distribution parameters under the given conditions;

[0014] The input to the state-value network is the current state s of the AUV. t The output is the value of this state under the current policy, which is a scalar.

[0015] Preferably, in step 2, the reward function r t for:

[0016] r t =r1+r2+r3

[0017] Where r1 represents the real-time reward, r2 represents the terminal reward, and r3 represents the angle-based penalty;

[0018] r1=(d l -d)*Ab

[0019] Where A and b are two hyperparameters used to control the size of the real-time reward; d l d represents the distance between the AUV and the dock at the previous update time; d represents the distance between the AUV and the dock at the current time.

[0020]

[0021] Where R is the hyperparameter of the reward function, and m is the difference between the attitude of the AUV when it arrives at the dock and its desired attitude; m max Let m be the maximum value, k be a constant, and sit A, sit B, sit C represent different terminal situations:

[0022] sit A:if d<ε d and|ψ-ψ d |<ε ψ and|θ-θ d |<ε θ

[0023] sit B:if d<ε d and|ψ-ψ d |<ε ψ or d<ε d and|θ-θ d |<ε θ

[0024] sit C:if d<ε d

[0025] Where, ε d , ε θ , ε ψ These are the distance threshold, pitch angle threshold, and heading angle threshold, respectively.

[0026] r3=e*|ψ l -ψ|+f*|θ l -θ|

[0027] Where, ψ l ψ and θ represent the heading angles of the AUV at the previous and current times, respectively. l θ and θ represent the pitch angle of the AUV at the previous moment and the pitch angle of the AUV at the current moment, respectively.

[0028] Preferably, the parameter update condition in step 3 is: the number of tuples in the experience pool is equal to the set hyperparameter.

[0029] Preferably, the dominant function in step 4 for:

[0030]

[0031] Where, δ t The time-series difference error at time t is represented by γ, the discount factor by λ, and a constant used to control the proportion of variance and bias by δ. t+1Characterizing the time series difference error at time t+1, δ T-1 Characterizes the time-series difference error at time T-1.

[0032] Preferably, step 4 utilizes temporal difference error to perform gradient descent on the parameters φ of the state-value network to achieve policy evaluation; including:

[0033] With (V) t trag -V φ (s t )) 2 Gradient descent is applied to the parameters φ of the state-value network as the loss function. M iterations are performed until the requirements are met, where V... t trag For the target state value network, V φ (s t () is a parameterized state-value network.

[0034] Preferably, step 5 introduces a rollback mechanism to perform gradient descent on the parameters θ of the policy network to achieve policy improvement; including:

[0035] With -L T (θ)-cS [π(θ)] (s t The loss function is used to perform gradient descent on the parameters θ of the policy network. After N iterations, when the requirement is met, S... [π(θ)] (s t Let be the entropy of the state distribution under the parameterized policy π, c represent the degree of exploration, and L be the entropy of the state distribution. T (θ) satisfies the following equation:

[0036] L T (θ)=E t [L(θ)]

[0037]

[0038] Among them, D KL (π old (π) represents the KL divergence between the old and new strategies, used to measure the degree of difference between the two strategies. t (θ) represents the rate of change of the policy, and α and δ are constants used to control the gradient of the loss function and the size of the clipping region, respectively.

[0039] Compared with the prior art, the advantages of the present invention are:

[0040] 1. The AUV dynamics model is adopted in a three-dimensional environment, which is closer to the actual physical model.

[0041] 2. A reward function suitable for the AUV docking problem was designed, the parameter update rule of the PPO algorithm was improved, and the convergence speed of the algorithm was accelerated. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of AUV docking in a 3D environment;

[0043] Figure 2 This is a flowchart of the AUV homing method based on deep reinforcement learning;

[0044] Figure 3 These are the reward function curves of the algorithm under different strategy evaluation and strategy improvement ratios;

[0045] Figure 4 shows the reward function curves of different algorithms under two environments, where the dock in Figure 4(a) is located at x d ,y d ,z d At position [100, 100, 50], the dock attitude is ψ. d ,θ d = [0,0], the dock in Figure 4(b) is located at x d ,y d ,z d At point [50, 100, -20], the dock attitude is ψ. d ,θ d = [π / 6, 0];

[0046] Figure 5 shows the motion trajectory of the AUV under two algorithms when the dock is stationary, where Figure 5(a) shows the dock located at x d ,y d ,z d At point [80, 80, 5], the attitude is ψ d ,θ d = [0,0], Figure 5(b) shows the dock located at x d ,y d ,z d = [60, 60, 10], attitude is ψ d ,θ d = [π / 2, 0];

[0047] Figure 6 shows the motion trajectory of the AUV under two algorithms when the dock moves in a straight line. In Figure 6(a), the initial position of the dock is located at x. d ,y d ,z d = [20, 60, 5] and moves in a straight line along the x-axis with a velocity component of 2 m / s. The initial position of the dock in Figure 6(b) is located at x. d ,y d ,z dAt point [30, 40, 10], it moves in a straight line along the y-axis at a speed of 1.5 m / s;

[0048] Figure 7 shows the motion trajectory of the AUV under two algorithms during the circular motion of the dock. In Figure 7(a), the initial position of the dock is located at x. d ,y d ,z d = [20, 60, 10], and moves clockwise in a curved path with a velocity component of 2 m / s and an angular velocity of 0.15 rad / s; Figure 7(b) shows the initial position of the dock at x d ,y d ,z d At the location [20, 60, 10], it makes counterclockwise circular motion on the plane with a speed of 1.5 m / s;

[0049] Figure 8 It is the motion trajectory of an AUV under certain special circumstances. Detailed Implementation

[0050] This invention aims to overcome the shortcomings of the aforementioned inventions by designing an AUV docking algorithm based on reinforcement learning in a three-dimensional environment. The problem is modeled using an unsimplified three-dimensional environment, and environmental information is directly mapped to rudder angle commands, thus considering the impact of AUV dynamics on the final planning result. Furthermore, existing reinforcement learning algorithms are improved, increasing their convergence speed and the optimality of feasible solutions.

[0051] 1. Environmental Modeling and Problem Description

[0052] This paper studies the docking problem of AUVs in a three-dimensional environment, as illustrated in the diagram below. Figure 1 As shown:

[0053] First, the dynamic model of the AUV in a three-dimensional environment is given:

[0054]

[0055] Where ΔG = GB represents the negative buoyancy of the AUV; ΔM xp Represents the unbalanced torque; m represents the mass of the AUV; (x c ,y c ,z c (J) represents the components of the AUV's centroid along the three axes in the volume coordinate system; xx J yy J zz () represents the moment of inertia of the AUV along its three axes in volume coordinates; T is the thrust; ρ is the density of seawater; S is the maximum cross-sectional area; L is the length; (x M ,y M ,z M(v) represents the position of the AUV in the Earth coordinate system; Mx ,v My ,v Mz ), (ω Mx ,ω My ,ω Mz ) represents the velocity and angular velocity components of the AUV along the three axes in volume coordinates; v M Represents the resultant velocity of the AUV; (θ) M ,ψ M ,φ M ) represents the pitch angle, yaw angle, and roll angle of the AUV; Represents the angle of attack and sideslip angle of an AUV; C xS , λ ij These are parameters related to fluid dynamics; These are parameters related to hydrodynamic torque; For normalized angular velocity, δ e ,δ r ,δ d These represent horizontal rudder angle, vertical rudder angle, and differential rudder angle.

[0056] The three-dimensional dock model is represented using kinematics, and its expression is similar to the last six equations above. It only requires changing the position of the dock (x... d ,y d ,z d ), the speed of the dock (v) dx ,v dy ,v dz ), the attitude of the dock (θ) d ,ψ d ,φ d ), the angular velocity of the dock (ω) dx ,ω dy ,ω dz Simply replace it with the appropriate option. In the simulation experiment below, it is assumed that the dock is stationary or moves according to certain rules (e.g., linear motion, circular motion, etc.).

[0057] Because AUVs operate at relatively low speeds during docking, they are significantly affected by ocean currents. Therefore, ocean currents must be considered during docking. This paper establishes a three-dimensional ocean current model as follows:

[0058]

[0059] In the above formula, v ox ,v oz ,v oy Characterizing the velocity components of ocean currents in a three-dimensional environment in the Earth coordinate system, which are composed of uniform ocean currents (the constant term A in the above equation). o Bo It is formed by superimposing the vortex term and the eddy term (the non-constant term in the above formula). Where τ represents the eddy intensity, p... center =[x center ,z center ] represents the center of the vortex in a two-dimensional plane, and ε represents the radius of the vortex. It is also worth noting that after incorporating the influence of ocean currents on the AUV, it is only necessary to replace the absolute velocity in the dynamic equations with the relative velocity.

[0060] The AUV docking problem can be described as follows: By designing a strategy π(a t |s t Control the AUV so that its relationship with the dry dock satisfies the following expression:

[0061]

[0062] In the above text, ε d , ε θ , ε ψ These represent the distance threshold, pitch angle threshold, and heading angle threshold, respectively. t a represents the state of the AUV at a certain time t. t Characterized in s t The next action to choose, π(a) t |s t ) characterizes state s t The AUV takes action a t The probability magnitude of s. t ,a t The specific expression will be given below.

[0063] 2. AUV Hopping Algorithm Based on Proximity Strategy Optimization

[0064] This invention utilizes the policy gradient algorithm in deep reinforcement learning to solve the AUV homing problem. The policy gradient algorithm directly parameterizes the policy π. θ (a t |s t The algorithm optimizes the policy by calculating the gradient of the policy parameters with respect to the performance metric. In the Proximal Policy Optimization (PPO) algorithm, the agent objective function is expressed as follows:

[0065]

[0066] In the above formula, The ratio of change in the characterization strategy ε represents the parameterization strategy updated in the last update; ε represents a pruning threshold used to control the difference between the new and old strategies. A dominance function is represented as follows:

[0067]

[0068] Where, δ t Characterizing the time-series difference error, δ t =r t +γV(s t+1 )-V(s t ), V(s t The state value function represents the state value function. It takes a state vector at a certain moment as input and outputs a scalar representing the value of a state under the current policy. In practical applications, the state value function is generally parameterized by φ, and therefore can be written as V φ (s t γ represents the discount factor, λ represents a constant used to control the proportion of variance and bias, and r t The reward received at time t represents the value of the reward. The remainder of this section will design a suitable action space, state space, and reward function for the AUV homing problem, and improve the PPO algorithm framework to enhance its performance.

[0069] 2.1 State Space Design

[0070] This paper designs the state space in the following form:

[0071] s t =[d,θ Δ ,ψ Δ ,θ t ,ψ t ,δ e ,δ r ,δ d ,Δv x ,Δv z ]

[0072] Where d represents the distance between the AUV and the dock, ψ Δ The heading angle of the AUV and the angle θ between the projection of the line connecting the AUV and the dock onto the horizontal plane are characterized by the heading angle of the AUV and the projection of the line connecting the AUV and the dock onto the horizontal plane. Δ The pitch angle of the AUV and the angle between the projection of the AUV and the line connecting the AUV and the dock onto the vertical plane are represented by the following formulas:

[0073]

[0074] Since AUVs need to meet certain terminal angle restrictions when docking, the difference between the AUV's attitude and the dock's attitude needs to be incorporated into the AUV's state space. The specific formula is as follows:

[0075]

[0076] Due to the influence of underwater ocean currents, AUVs need to have the ability to adaptively resist ocean currents. Therefore, ocean current information also needs to be encoded into the state required by the algorithm. The specific formula is as follows:

[0077]

[0078] In the above formula, v Mx0 ,v Mz0 The velocity components of the AUV in the world coordinate system are characterized and calculated as follows:

[0079]

[0080] Finally, due to the significant difference in the orders of magnitude of the aforementioned physical quantities, a weight matrix A is needed to weight the state matrix to obtain the final state vector:

[0081] s t ←s t ·A

[0082] Where A represents the weight matrix, A = [a1, a2, ... a2] n ], n represents the size of the dimension of the state space.

[0083] 2.2 Motion Space Design

[0084] Since this paper adopts an "end-to-end" control method, the motion space is designed as follows:

[0085]

[0086] In the above formula, These respectively characterize the horizontal, vertical, and differential rudder angle coefficients. The input to the policy network is the current state s of the AUV. t The output is the strategy action a. t In state s t The probability distribution parameters under given conditions (e.g., the mean and variance of a Gaussian distribution can be output).

[0087] After the strategy network outputs the above actions, they need to be converted into actual rudder angles using the following formula:

[0088]

[0089] Where, δ emax ,δ rmax ,δ dmax These represent the maximum permissible rudder angle under actual conditions. It's also worth noting that a first-order model is used in the algorithm simulation, meaning the output action does not represent the actual angle of the rudder.

[0090]

[0091] Where K represents a proportionality constant, δ Δmax The value represents the largest change in rudder angle within a time step, and clip(x,a,b) represents clipping x to the interval [a,b]. Intuitively, the larger the difference between the rudder angle output by the strategy and the current actual rudder angle, the larger the change in rudder angle.

[0092] 2.3 Reward Function Design

[0093] The reward function, as a crucial metric within the reinforcement learning framework, is key to the success of the final algorithm. Since the terminal needs to meet certain constraints during homing, this section designs the reward function in the following form:

[0094] r t =r1+r2+r3

[0095] Where r1 represents the real-time reward of the algorithm, r2 represents the terminal reward of the algorithm, and r3 represents the angle-based penalty term. This paper represents the real-time reward in the following form:

[0096] r1=(d l -d)*Ab

[0097] In the above formula, A and b represent two hyperparameters used to control the size of the real-time reward, and d l The distance between the AUV and the dock at the previous update time is represented by d; the distance between the AUV and the dock at the current time is represented by d. When the AUV approaches the dock at a certain time, it receives a positive reward; conversely, it receives a negative reward. When designing the hyperparameters, the value of hyperparameter b cannot be too large. If the value is too large, it will overwhelm the positive guidance signals, preventing them from achieving the objective.

[0098] Typically, the terminal reward of an algorithm is a fixed value, with a large positive reward for completing the current task and a small negative reward for failing to complete the task. This reward structure is ineffective when terminal constraints are difficult to satisfy. Therefore, this paper smooths the terminal reward function to guide AUVs in the desired direction.

[0099]

[0100] In the above formula, R represents the hyperparameter of the reward function, and m represents the difference between the attitude of the AUV when it arrives at the dock and the desired attitude, expressed as:

[0101] m=(|ψ-ψ d |+|θ-θ d |)

[0102] m max Characterizes a constant and m max >max(|ψ-ψ d |+|θ-θ d |), k represents a constant used to control the magnitude of the second term. sit A, sit B, sit C represent different terminal cases, listed below:

[0103] sit A:if d<ε d and|ψ-ψ d |<ε ψ and|θ-θ d |<ε θ

[0104] sit B:if d<ε d and|ψ-ψ d |<ε ψ or d<ε d and|θ-θ d |<ε θ

[0105] sit C:if d<ε d

[0106] The motivation for setting the terminal reward to the above reward function is to provide some guidance to the AUV when it approaches the dock but does not enter the dock at the specified angle, so that the AUV does not receive any feedback when it does not meet all constraints.

[0107] The third part of the reward function is the angle penalty term, expressed as follows:

[0108] r3=e*|ψ l -ψ|+f*|θ l -θ|

[0109] In the above formula, ψ l ψ and θ represent the heading angle of the AUV at the previous moment and the heading angle of the AUV at the current moment, respectively. l θ and θ represent the pitch angle of the AUV at the previous moment and the pitch angle of the AUV at the current moment, respectively. The motivation for adopting the above reward function is to expect the planned trajectory to have a certain degree of smoothness.

[0110] 2.4 Strategy Parameter Update Method

[0111] This section improves the PPO algorithm, forming an improved proximal policy optimization algorithm (IPPO). Since the PPO algorithm is based on the belief region policy optimization (TRPO) algorithm, its convergence guarantee rests on a relatively small policy update ratio. However, limiting the update ratio only on a single sample cannot guarantee limiting the update ratio globally, because the pruning method of the PPO algorithm does not penalize samples exceeding the update ratio. Therefore, this paper introduces a "rollback" mechanism to correct the above problem:

[0112] L T (θ)=E t [L(θ)]

[0113]

[0114] In the above formula, D KL (π old ,π) represents the KL divergence between the old and new strategies, used to measure the degree of difference between the two strategies, and δ represents a constant threshold used to control the size of the clipping region. t (θ) represents the rate of change in the strategy. Intuitively, when the differences between the two strategies are significant and pruning is necessary, unlike the original PPO algorithm, it generates a certain gradient to curb r. t (θ) is further increased, thereby better controlling the differences between the old and new strategies.

[0115] In addition, to further improve the efficiency of the algorithm, this paper divides the update process into two parts: policy evaluation and policy improvement. By adjusting the ratio of policy evaluation to policy improvement, the fluctuations during training can be mitigated.

[0116] In summary, the AUV homing algorithm based on deep reinforcement learning has been introduced, and the pseudocode of the algorithm is listed below:

[0117] Table 1 Algorithm Pseudocode

[0118]

[0119]

[0120] In the pseudocode above, K, M, and N represent the number of iterations; S represents the entropy of the strategy distribution; c represents the degree of exploration; S [π(θ)] (s t Let π be the entropy of the state distribution under the parameterized policy π. S max ,T max These represent the maximum number of rounds and the maximum number of steps per round, respectively. Store p(a) t This is to calculate the strategy ratio described above. To distinguish between the probability distribution function and the value of the probability distribution function corresponding to a specific action, p(a) is used. t ) characterizes state s t Downsampling to a t The probability is used to replace π in the above formula. θ (a t |s t To ensure the algorithm gradually stabilizes, in actual deployment, c = c0e. -at . Figure 2 A simplified flowchart of the method is provided.

[0121] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0122] 3. Algorithm Simulation

[0123] To verify the effectiveness of the proposed algorithm, this paper simulates the algorithm on a Windows 11 + Python 3 + PyTorch platform. Since the ratio of policy updates to policy improvements needs to be adjusted during algorithm design, some hyperparameters required by the algorithm need to be determined in a simple environment before simulation experiments are conducted. The simulation results are as follows. Figure 3 As shown.

[0124] Figure 3 In the diagram, the horizontal axis represents the number of training rounds, and the vertical axis represents the reward sum. The simulation environment is as follows: the AUV is located at the origin, and the dock is located at x... d ,y d ,z d At position [100, 100, 50], the desired attitude is ψ. d ,θ d = [0,0]. The curve above was obtained through 50 steps of smoothing. It is easy to see that the algorithm performs better when M=8 and N=6. Therefore, the above hyperparameters were used in subsequent simulations.

[0125] To verify the performance of the above algorithm, this paper compares it with the original PPO algorithm and the SAC algorithm, as shown in Figure 4.

[0126] In Figure 4, the environments in both figures are located at the origin. The dock in Figure 4(a) is located at x. d ,y d ,z d = [100, 100, 50]; the dock in Figure 4(b) is located at x d ,y d ,z d = [50, 100, -20]; the dock attitude in Figure 4(a) is ψ d ,θ d= [0,0], the dock attitude in Figure 4(b) is ψ d ,θ d = [π / 6, 0]. The three algorithms were subjected to three Monte Carlo simulations in the same environment. The solid line in the figure represents the mean, and the colored part represents the standard deviation. Observing the figure above, it is not difficult to see that the improved IPPO algorithm performs better in the two environments mentioned above. Although the PPO algorithm and SAC algorithm can sometimes complete the basic docking task, they still suffer from unstable training and suboptimal paths.

[0127] To ensure the algorithm can successfully dock the AUV in most scenarios, the location of the dock and the strength of the ocean current need to be randomly initialized and then trained. During training, if the algorithm fails to complete the current task, it continues learning in the next scenario; if the algorithm completes the current task, it randomly initializes a new task in the next scenario and begins learning. The main hyperparameters of the algorithm are listed below:

[0128] Table 2 Algorithm Hyperparameters

[0129]

[0130] Subsequently, this paper will demonstrate the performance of each algorithm in different environments, as follows:

[0131] 1. The dock is not stationary at the same time:

[0132] In Figure 5, Figure 5(a) represents the location of the dock at x. d ,y d ,z d At point [80, 80, 5], the attitude is ψ d ,θ d = [0,0]. Figure 5(b) represents the location of the dock at x d ,y d ,z d = [60, 60, 10], attitude is ψ d ,θ d = [π / 2, 0]. The second graph in each row represents the change in distance between the AUV and the dock over time; the third graph in each row represents the change in the sum of the absolute values ​​of the differences between the AUV attitude and the dock attitude over time. Since it has been explained earlier that the PPO algorithm is an improvement on the IPPO algorithm, the PPO algorithm will not be included in this comparison in this section or below. Observing the two graphs above, it is easy to see that the improved IPPO algorithm has better performance than the SAC algorithm in both docking time and error.

[0133] 2. When the dock is moving in a straight line:

[0134] In actual navigation operations, AUVs may dock with mother ships sailing at sea. In this case, docking with a moving dock is particularly important. Based on this, this paper simulates the process, and the simulation results are shown in Figure 6.

[0135] In Figure 6, the green line represents the slowly moving dock. In Figure 6(a), the initial position of the dock is located at x. d ,y d ,z d = [20, 60, 5] and moves in a straight line along the x-axis with a velocity component of 2 m / s. In Figure 6(b), the initial position of the dock is located at x d ,y d ,z d At point [30, 40, 10], it moves in a straight line along the y-axis at a speed of 1.5 m / s. In both diagrams, the dock's attitude is ψ. d ,θ d =[0,0]. It is not difficult to see from the above two sets of pictures that when the dock moves in a straight line, the performance difference between the two algorithms is not significant. However, the IPPO algorithm has certain advantages in terms of the terminal distance and terminal angle difference.

[0136] 3. When the dock is moving in a curved path:

[0137] The aforementioned simulations were all conducted with the terminal angle remaining constant. This section will simulate the scenario where the terminal angle is constrained to change over time, as follows:

[0138] In Figure 7, the green line represents the slowly moving dock. In Figure 7(a), the initial position of the dock is located at x. d ,y d ,z d = [20, 60, 10], and moves clockwise in a curved path with a velocity component of 2 m / s and an angular velocity of 0.15 rad / s; in Figure 7(b), the initial position of the dock is located at x d ,y d ,z d At the location [20, 60, 10], it makes counterclockwise circular motion on the plane at a speed of 1.5 m / s. In the two figures, the attitude of the dock changes uniformly with time. It is not difficult to see from the two sets of figures above that when the dock is moving in a curved path, the IPPO algorithm has certain advantages in the two physical quantities of terminal distance and terminal angle difference.

[0139] 4. Special Circumstances

[0140] In some extreme cases, the dock may be located within the minimum turning radius of the AUV. In this case, turning the rudder fully in the corresponding direction will not complete the docking task, but the designated dock can be found through other means. The specific simulation is as follows: Figure 8 In the middle, the initial position of the AUV dock is x. d ,y d ,z d At the location [10-10, -2], the initial attitude is ψ. d ,θ d =[0,0]. Due to the short distance between the AUV and the dock, the AUV cannot directly enter the dock. However, a reinforcement learning-based algorithm can find a "retreat-to-advance" strategy, first moving away from the dock a certain distance, and then approaching the dock. This allows the AUV to complete the docking task even in relatively special environments.

[0141] Overview:

[0142] This invention discloses an AUV docking method based on deep reinforcement learning. First, the invention establishes a dynamic model of the AUV and a kinematic model of the dock in a three-dimensional environment, and introduces the influence of ocean currents during the docking process. Second, based on the above models, the invention designs suitable state and action spaces. Third, to enable the algorithm to make fuller use of the terminal state information of the AUV and the dock, the invention designs a new reward function, providing guidance even when the AUV has not completed the docking task. Finally, the invention is based on the Proximal Policy Optimization (PPO) algorithm in deep reinforcement learning and improves it to form an improved Proximal Policy Optimization (IPPO) algorithm. This method introduces a "trust domain rollback mechanism" to keep the policy change ratio at a small value, while adjusting the ratio of policy evaluation and policy improvement to stabilize the training algorithm. Simulation results at the end of the paper verify that the improved algorithm has a faster convergence rate than the unimproved algorithm and can complete the docking task in both static and moving dock conditions.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for AUV hull docking based on deep reinforcement learning, comprising: Step 1: Real-time collection of the current state of the AUV based on sonar and self-inertial navigation information , using the current parameterized policy network Sample actions , and record the corresponding probability density value in the policy distribution as ;​ Step 2: Take action Substitute into the AUV dynamics equation and combine with the dock state to calculate the state at the next time And the reward function , the tuple is stored in the experience pool; the reward function is: ； wherein, represents a real-time reward, represents a terminal reward, represents an angle-based penalty term; ； wherein, are two hyperparameters used to control the size of the real-time reward; is the distance between the AUV and the dock at the last update time; is the distance between the AUV and the dock at the current time. ； wherein, is a hyperparameter of the reward function, is the difference between the AUV's attitude and the desired attitude when the AUV reaches the dock; is the maximum of m is a constant, is a constant, characterizes different terminal conditions: ； ； ； wherein, , , are a distance threshold, a pitch angle threshold and a heading angle threshold, respectively; ； wherein, and and and and Step 3: When the number of tuples in the experience pool meets the parameter update condition, step 4 is performed, otherwise, go to step 1; Step 4: Extract the current state from the experience pool tuple data. The state at the next moment The input is fed into the state-value network to obtain the state-value function value, which is then used to calculate the advantage function. Sampling from the experience pool Each tuple utilizes temporal difference error to evaluate the parameters of the state value network. Perform gradient descent to evaluate the policy; Step 5: Sample again from the experience pool a tuple, introducing a rollback mechanism to the parameters of the policy network perform gradient descent, achieving policy improvement; Step 6: When the AUV meets the termination condition, end, otherwise, go to step 1.

2. The deep reinforcement learning based AUV hull homing method of claim 1, wherein, The policy network and the state value network are both independent feedforward neural networks, wherein the input of the policy network is the current state of the AUV , and the output is a policy action The probability distribution parameters under the condition of the state ​ The input to the state-value network is the current state of the AUV and the output is the value of this state under the current policy, which is a scalar.

3. The deep reinforcement learning based AUV hull homing method of claim 1, wherein, The parameter update condition in step 1 is that the number of tuples in the experience pool is equal to the set hyperparameter.

4. The deep reinforcement learning based AUV hull homing method of claim 1, wherein, The merit function in step 4 is: ； wherein, characterized the timing difference error at the time instant, characterized a constant for controlling the ratio of variance and bias, characterized the timing difference error at the time instant, characterized the timing difference error at the time instant.

5. The deep reinforcement learning based AUV hull homing method of claim 1, wherein, the parameters of the state value network using the timing difference error in step 4 performing gradient descent to implement policy evaluation; comprising: As gradient descent on the parameters of the state value network with the loss function is performed M times, where is the target state value network, is the parameterized state value network.

6. The deep reinforcement learning based AUV hull homing method of claim 1, wherein, the step 5 introduces a rollback mechanism to the parameters of the policy network performing gradient descent to achieve policy improvement; Comprising: With the parameters of the policy network as loss function gradient descent is performed, until a requirement is met, which has been iterated N times, wherein is the entropy of the state distribution under the parameterized policy characterizes the size of the exploration degree, satisfies the following equation:​ ； ； wherein, KL divergence representing the new and old policies, for measuring the difference between the two policies, a policy change ratio, are constants for controlling the gradient of the loss function and the clipping region size, respectively.

Images