Vector-propelled AUV path planning method based on deep reinforcement learning

Abstract

The present invention discloses a vector propulsion AUV path planning method based on deep reinforcement learning, comprising: step 1) collecting the current state s of the AUV in real time based on sonar and its own inertial navigation information; t and action a t , input policy network π θ (a t |s t ), output strategy action a t , realize AUV path planning, and use reward function to calculate reward value r t , use the AUV dynamics equation to calculate the next state s t+1 , the tuple [s t ,a t ,r t ,s t+1 ] is stored in the experience pool. Step 2) When the number of tuples in the experience pool is greater than the set value N, M tuples are sampled from it; go to step 3); otherwise, go to step 1); step 3) Under the maximum entropy reinforcement learning framework, the policy network parameters θ, the evaluation network parameters and the temperature parameter α are updated respectively; step 4) When the AUV meets the termination condition, it ends, otherwise, go to step 1).

Description

Technical Field

[0001] The present invention belongs to the technical field of AUV path planning, and in particular relates to a vector-propelled AUV path planning method based on deep reinforcement learning. Background Art

[0002] Autonomous underwater vehicles (AUVs) are essential tools for ocean exploration. As the difficulty of AUV missions increases, higher demands are placed on their maneuverability and autonomy. As one of the core technologies for achieving AUV autonomy, path planning has attracted the attention of numerous researchers in recent years.

[0003] Path planning refers to planning a path from a starting point to a target point under certain constraints (such as dynamic constraints, environmental constraints, etc.), and meeting certain performance requirements (such as the shortest planning distance, the shortest planning time, etc.). There are related patents that have studied the path planning problem of AUV. For example, the Chinese patent application with publication number CN118857320A proposes a graph-based path planning algorithm, which uses a bidirectional A * The algorithm determines the local path point, which performs a two-way search for the starting point and the target point, and the traditional A * The algorithm improves the search efficiency. Subsequently, DWA (dynamic window algorithm) is used to connect local path points to avoid unknown obstacles in the environment map. The Chinese patent application with publication number CN118838357A proposes a sampling-based AUV path planning algorithm to solve the two-dimensional path planning problem in the ocean current environment. It uses a fast-expanding random tree algorithm to plan a low-energy path, and by integrating ocean current information during random sampling, the algorithm can quickly reduce the energy consumption of the AUV with the help of ocean currents. The Chinese patent application with publication number CN115202356B proposes an AUV path planning algorithm based on a heuristic optimization method. Based on the kinematic model of under-actuated AUV in a three-dimensional environment, it gives the hard constraints and soft constraints of the path planning problem, and searches for the optimal path through an improved ant colony algorithm. The improved algorithm uses adaptive parameters to prevent the algorithm from falling into a local optimum.

[0004] With the rapid development of deep reinforcement learning (DRL) technology, researchers are increasingly using deep reinforcement learning technology to solve AUV path planning problems. The advantage of using deep reinforcement learning technology is that this type of method learns by interacting with the environment and has strong environmental adaptability. At the same time, compared with heuristic-based optimization strategies, DRL pays more attention to the internal structure of the strategy. In addition, the powerful representation ability of neural networks enables it to solve more complex planning and control problems. For example, the Chinese patent application with publication number CN118394089A uses the DQN algorithm in DRL technology to solve the high-latitude AUV path planning problem. It integrates terrain information, inertial guidance error information, and target point information into the reward function, allowing the AUV to reach the target point faster. The Chinese patent application with publication number CN113064422A also uses the DQN algorithm to solve the path planning problem, but when sampling from the experience pool, it uses priority experience replay technology to speed up the convergence of the algorithm. The Chinese patent application with publication number CN115790608A uses the SAC algorithm in DRL technology to solve the AUV path planning problem in a three-dimensional environment with ocean currents. In order to make the training model more in line with the actual environment, it uses actual ocean current data to model the ocean currents based on the Gauss-Markov model, and trains the algorithm in the modeled environment to improve the robustness and environmental adaptability of the algorithm.

[0005] Through the investigation of the above-mentioned existing technologies, it was found that traditional algorithms (graph-based algorithms, sampling-based algorithms, and heuristic optimization-based algorithms) have more or less the problems of large computational complexity and poor generalization performance, and most of them model the problem at the kinematic level, ignoring the impact of the AUV's dynamics on the algorithm. In addition, the planning environment of some algorithms is a two-dimensional grid, which means that after the planning is completed, it is necessary to use some method (such as cubic spline method) to smooth the planned path. Although the algorithm based on deep reinforcement learning has a certain degree of environmental adaptability, there is still room for optimization in the convergence speed of the algorithm. In addition, the control input of most patented AUV path planning methods is relatively simple, only the rudder deflection angle, which will make its maneuverability poor. However, there is little research on the path planning problem of AUV using a combined control method of vector thrusters and rudders, and this type of control method can effectively improve the maneuverability of AUVs.

[0006] Unlike conventional single-control variable path planning problems, the joint control method requires a balance between the usage ratio of the two to avoid suboptimal paths caused by excessive use of a certain control variable, which brings new challenges to the path planning problem. Summary of the Invention

[0007] In view of this, the present invention proposes a vector-propelled AUV path planning method based on deep reinforcement learning, comprising:

[0008] Step 1) Real-time acquisition of the AUV’s current state s based on sonar and its own inertial navigation information t , according to the parameterized strategy network π θ (a t |s t ) Output action a t , use the reward function to calculate the reward value r t , use the AUV dynamics equation to calculate the next state s t+1 , the tuple [s t ,a t ,r t ,s t+1 ]Deposit into experience pool

[0009] Step 2) When the experience pool When the number of tuples in the Sample M tuples; go to step 3); otherwise, go to step 1);

[0010] Step 3) Under the maximum entropy reinforcement learning framework, the policy network parameters θ and the evaluation network parameters And the temperature parameter α is updated;

[0011] Step 4) When the AUV meets the termination condition, the process ends; otherwise, go to step 1).

[0012] Preferably, the step 1) further comprises:

[0013] Real-time acquisition of the AUV's current status based on sonar and its own inertial navigation information t 、Action a t , and the state s at the next moment t+1 ;

[0014] According to the state s t 、Action a t , the current reward r is calculated by the reward function t ;

[0015] The tuple [s t ,a t ,r t ,s t+1 ]Deposit into experience pool

[0016] Continue the above steps until the experience pool The number of tuples in is greater than the set value N.

[0017] Preferably, the policy network and the evaluation network are two independent neural networks; the layers of each neural network are connected using batch normalization and bounded activation functions; the action distribution is modeled using Gaussian distribution, and the mean and standard deviation of the Gaussian distribution are finally output.

[0018] Preferably, the step 1) rewards a value r t for:

[0019] r t =r z +r s +r d

[0020] Among them, r z Represents the terminal reward, giving a larger positive feedback when reaching the target point and a smaller negative feedback when colliding with an obstacle; r s Represents immediate reward, r d Characterizes rewards related to rudder angle and geometric deflection angle.

[0021] Preferably, the step 3) comprises:

[0022] Step 3-1) The loss function is used to evaluate the network Parameters Make updates;

[0023] Step 3-2) J π (θ) is the loss function for the policy network π θ (a t |s t )’s parameters θ are updated;

[0024] Step 3-3) Update the temperature parameter α using J(α) as the loss function.

[0025] Preferably, the loss function in step 3-1) for:

[0026]

[0027] Where, |·| sg Its gradient is not required for characterization, represents the parameterized evaluation network, a t+1 Indicates the next action, s t+1 Indicates the state at the next moment, q t ,q t+1 Respectively represent when the input of the evaluation network is [s t ,a t ],[s t+1 ,a t+1], γ represents the discount factor, γ∈[0,1].

[0028] Preferably, the step 3-2) loss function J π (θ) is:

[0029]

[0030] Where D KL (·||·) is used to measure the difference between the two strategies, π θ (·||s t ) represents the state s t The strategy distribution under Representation in state s t The non-normalized probability distribution of the action-state value function under , Characterizes the normalization constant.

[0031] Preferably, the loss function J(α) in step 3-3) is:

[0032]

[0033] Where, Characterizes a constant, π θ (a t |s t ) is the policy network. Represents an expected symbol, the action inside the expected symbol a t Using parameterized strategy π θ Take samples.

[0034] Preferably, the termination condition of step 4) includes: the actual number of running scenes is greater than the maximum number of running scenes super parameter T me .

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] 1. This paper uses deep reinforcement learning technology to solve the path planning problem of vector-propelled AUVs. It models the path based on the dynamics of vector-propelled AUVs and adopts an "end-to-end" control method. By fully leveraging the characteristics of deep reinforcement learning technology, the planned strategy has strong environmental adaptability.

[0037] 2. Based on the characteristics of the problem, the present invention designs the state space, action space and reward function that fit the path planning problem under joint control, so that the algorithm converges faster.

[0038] 3. For the SAC algorithm, this paper introduces an adaptive temperature parameter, eliminates the target evaluation network, and adds a batchnorm architecture, which increases the sample efficiency of the algorithm and reduces the convergence time of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the path planning of the vector-propelled AUV;

[0040] Figure 2(a) shows the paths under different propulsion modes, and Figure 2(b) shows the relative power required by the three to achieve the same speed.

[0041] Figure 3(a) is a schematic diagram of the AUV sonar model, and Figure 3(b) is an example of sonar detection data;

[0042] Figure 4 is a flow chart of the method of the present invention;

[0043] Figure 5 This is a graph showing the total reward change trend of the ISAC algorithm of the present invention;

[0044] Figure 6 is the path planned by the algorithm in environment 1;

[0045] Figure 7 is the rotation ratio of the rudder angle and the vector angle in environment 1;

[0046] Figure 8 is the path planned by the algorithm in environment 2;

[0047] Figure 9 is the rotation ratio of the rudder angle and the vector angle in environment 2;

[0048] Figure 10 is the path planned by the algorithm in environment 3;

[0049] Figure 11 is the rotation ratio of the rudder angle and the vector angle in environment 3. DETAILED DESCRIPTION

[0050] To address these issues, this paper proposes a path planning method for vector-propelled AUVs based on deep reinforcement learning. First, the method controls the AUV to reach a specified target point by directly mapping the AUV's state space to the rudder and vector thruster angles. The action space is modeled as a continuous space, thus avoiding subsequent path smoothing steps. Second, the reward function increases the penalty for using vector thrusters, taking into account the turning characteristics of rudder and vector angles. Finally, the method optimizes the SAC algorithm in DRL technology to accelerate its convergence.

[0051] The present invention aims to solve the path planning problem of vector-propelled AUV in marine environment. The path planning diagram is as follows Figure 1 shown.

[0052] Figure 1 XOZ, Xu O u Z u The blue sector represents the sonar detectable range, and the rotation angle from the X-axis to the Z-axis is defined as a positive angle.

[0053] An AUV with vector propulsion in a two-dimensional environment can be modeled as follows:

[0054]

[0055] In the above formula, x c Represents the center of gravity in the hull coordinate system at X u The coordinate components of the axis. x0, z0, ψ represent the orientation and attitude of the AUV in the world coordinate system, v x ,v z ,ω y represents the speed and angular velocity of the AUV in the hull coordinate system, ρ represents the water density, S represents the maximum cross-sectional area of ​​the AUV, m represents the mass of the AUV, and I yy represents the moment of inertia of the AUV, L represents the length of the AUV, and λ 11 ,λ 33 ,λ 35 ,λ 55 ,C xS , Represents parameters related to fluid dynamics, Represents the parameter related to the fluid dynamic torque, δ r represents the vertical rudder angle, and A(ξ) represents the thrust correction coefficient. ξ represents the geometric deflection angle of the vector thruster. When the vector thruster nozzle deflects, thrust loss occurs, so thrust correction is required:

[0056]

[0057] The introduction of vector thrusters will reduce the turning radius of the AUV and improve its maneuverability, but the thrust loss will reduce the speed of the AUV when turning. Figure 2(a) shows the navigation routes and travel distances of the three cases when the rudder angle and vector geometric angle take the maximum value under the same running time. Figure 2(b) shows the relative power required to achieve the same speed under the three propulsion modes (expressed as (These represent different propulsion modes.) Observations show that while vector propulsion reduces the AUV's turning radius, it requires more power to achieve the same speed, resulting in higher energy consumption for the same distance. Therefore, when employing dual control using both the vector nozzle and rudder, a trade-off must be made between their use, ensuring the AUV reaches its target as quickly as possible while optimizing energy consumption.

[0058] The sonar model in this paper adopts a ray model. Figure 3(a) is a schematic diagram of the AUV sonar model, and Figure 3(b) is an example of sonar detection data. Where R represents the maximum detection radius of the sonar, Θ represents the maximum detection angle of the sonar, and d = [d1, d2, ..., d i ,…],0 <i≤i max Represents the distance between the sonar and the obstacle in the i-th direction.

[0059] Based on the above model and the vector geometry angle turning characteristics, this paper can establish the following path planning problem: by adjusting the rudder angle and vector geometry angle, and combining sonar perception information, the AUV is guided to avoid obstacles and reach the target with the minimum energy consumption E. Therefore, it can be written as follows:

[0060]

[0061] subject to:

[0062]

[0063] In the above formula, π θ (δ r ,ξ|s) represents the AUV executing δ in state s r ,ξ probability, that is, strategy; x0(t), z0(t) represent the coordinates of AUV in the world coordinate system at time t; x g ,z g represents the target coordinate; x ob,i ,z ob,i ,r ob,i represents the coordinates and radius of the i-th obstacle; T max ,δ max ,ξ max ,ζ represents the time limit, rudder angle limit, geometric deviation angle limit and target threshold.

[0064] The above path planning problem can be modeled as a Markov decision process (MDP), which uses the tuple<S,A,P,R,P0,γ,H> Indicates that S represents the state space, A represents the action space, and P represents the state transfer function P(s t+1 |s t ,a t ), R represents the reward function, P0 represents the initial state distribution, γ represents the discount factor, γ∈[0,1], and H represents the length of each episode. For MDP, the objective function of reinforcement learning can be expressed as:

[0065]

[0066] where π θ (a t |s t) represents a strategy parameterized by θ. This paper uses a maximum entropy reinforcement learning framework to solve the path planning problem of a vector-propelled AUV, giving the AUV a certain exploration capability. The objective function in this framework can be written as follows:

[0067]

[0068] In the above formula, α represents a hyperparameter, H(·|s t ) represents the state s t This paper will use the SAC (soft-actor-critic) algorithm to optimize the policy parameter θ and improve it to speed up the convergence of the algorithm.

[0069] This invention first establishes the dynamic equations of the vector-propelled AUV in the marine environment, and the simulation verifies that the vector thruster can improve the maneuverability of the AUV, but at the same time it will also increase the energy loss of the AUV; secondly, based on the above dynamic model, the present invention proposes a state space and action space adapted to this problem, and according to the characteristics of the vector thruster, increases the penalty for the use of the vector thruster in the reward function to control the unnecessary energy consumption caused by excessive use of the vector thruster; finally, the present invention uses the "soft actor-critic" algorithm (SAC) as a framework to update the policy network parameters, and improves it to accelerate the convergence speed of the algorithm. The simulation at the end of the article verifies that the improved SAC algorithm (ISAC) of the present invention can better balance the use ratio of the rudder angle and the vector deviation angle in a variety of environments to complete the path planning task.

[0070] The vector-propelled AUV path planning method based on deep reinforcement learning in this invention adopts a vector-propelled AUV path planning algorithm based on SAC, which is specifically introduced as follows:

[0071] The following will introduce the state space, action space, reward function and parameter update method required by the SAC algorithm.

[0072] This paper designs the state space into two parts, namely the state information of the spacecraft itself and sonar to sense external information

[0073]

[0074] Among them, δ r (t), ξ(t) represents the rudder angle and vector deviation at time t; d(t) represents the sonar perception data at time t; D represents the distance between the current initial point under the curtain and the target; the motivation for dividing each state component by a constant is to map all state components to the [-1,1] interval, which helps training generalization. Δ (t),ψΔ (t) represents the distance between the AUV and the target and the line of sight angle, which is expressed as follows:

[0075]

[0076]

[0077] Therefore, the final state vector can be expressed as follows:

[0078]

[0079] This paper adopts an "end-to-end" control method to directly map the state space into the action space. Therefore, this paper models the action space as follows:

[0080]

[0081] in, The proportional coefficient representing the rudder angle and the geometric deflection angle of the vector thruster can be transformed as follows to obtain the actual output action:

[0082] a t ←a t ·[δ max ,ξ max ]

[0083] In order to accelerate the convergence of the algorithm, this paper designs the reward function as follows:

[0084] r t =r z +r s +r d

[0085] Among them, r z Representation terminal rewards:

[0086]

[0087] That is, a larger positive feedback is given when the target point is reached, and a smaller negative feedback is given when the obstacle collides with it. s Representation 1 instant reward:

[0088] r s =a(d Δ (t-1)-d Δ (t))-b

[0089] d Δ (t-1),d Δ(t) represents the distances of the AUV from the target point at the previous moment and the current moment. The motivation for expressing it in the above form is to reward actions that bring the AUV closer to the target point, and a negative number (i.e., the hyperparameter b in the above formula) is added at each time step to control the AUV to reach the target point as soon as possible. r d Represents a reward related to the rudder angle and geometric deviation angle:

[0090] r d =-c(|δ r (t - 1)-δ r (t)|+|δ r (t)|)-d(|ξ(t - 1)-ξ(t)|+|ξ(t)|)

[0091] In the above formula, δ r (t - 1) and ξ(t - 1) represent the rudder angle and geometric deviation angle at the previous moment respectively. The motivation for adopting the above reward function is to make the control quantity as smooth as possible. In addition, in the setting of hyperparameters, c < d should be taken because, as mentioned before, using vector propulsion will reduce the speed of the AUV. Therefore, the reward function needs to increase the penalty for using vector propulsion.

[0092] The SAC algorithm requires parameterizing the action-value function network and the policy network π θ (a t |s t ) and guiding the algorithm to converge to the optimal policy by updating the parameters of the above two networks. The specific update strategy is as follows: The action-value function network is updated using the following loss function:

[0093]

[0094] where represents the target value function network, which is used to alleviate the unstable performance during the training process. The update method is as follows:

[0095]

[0096] In the above formula, τ represents a hyperparameter used to control the update speed of the target value network. In the update, in order to reduce the overestimation bias of the Q value, a double Q-network architecture is used. The policy network π θ (a t |s t ) is updated using the following loss function:

[0097]

[0098] In the above formula Represents a normalization constant. In order to reduce the variance during training, the SAC algorithm uses the reparameterization technique:

[0099] a t =f θ (ε t ;s t )

[0100] In the above formula, ε t Represent an input noise vector, which is sampled from a fixed distribution (such as a Gaussian distribution with a mean of 0 and a variance of 1), and bring it into the policy network π θ (a t |s t ) loss function, we can get:

[0101]

[0102] Unlike traditional reinforcement learning algorithms, the maximum entropy reinforcement learning framework introduces a temperature parameter α. Generally, α can be treated as a fixed value, but this approach will cause the strategy to maintain the same exploration in all state spaces. Therefore, the SAC algorithm improves the above problem by adaptively adjusting the temperature parameter:

[0103]

[0104] in, Represents a constant, which is usually initialized to the opposite of the action dimension in practical applications.

[0105] However, the above SAC algorithm still has room for improvement in terms of convergence speed. Based on this, this paper introduces the CrossQ algorithm, which improves the algorithm in the following two aspects:

[0106] First, the algorithm adopts a wider network architecture, that is, the number of hidden layer neurons is increased; second, it deletes the target network architecture and uses batch normalization technology to stabilize the algorithm training and achieve the above results. The improved proxy loss function is as follows:

[0107]

[0108] Among them, |·| sg The characterization does not require its gradient. In summary, the vector propulsion AUV path planning pseudo code based on the ISAC algorithm of the present invention can be given as shown in Table 1:

[0109] Table 1. Vector propulsion AUV path planning algorithm based on ISAC

[0110]

[0111] In the above table, N represents When the number of tuples in reaches N, the subsequent parameters are updated. M represents the number of samples to be sampled from the experience pool for each update. me , T mt Respectively represent the maximum number of episodes the algorithm runs and the maximum number of running steps for each episode.

[0112] The method flow chart of the present invention is as follows Figure 4 shown. Figure 4 The two networks in the middle and lower parts represent the evaluation network and the action network. The layers are connected using batch renormalization (BRN) and a bounded activation function (sigmoid). The action distribution is modeled using a Gaussian distribution, and the mean and standard deviation of the Gaussian distribution are finally output.

[0113] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0114] Example

[0115] The embodiment of the present invention provides a vector-propelled AUV path planning method based on deep reinforcement learning. The vector-propelled AUV path planning algorithm is simulated using python3+pytorch. The hyperparameters of the algorithm are shown in the following table, and the specific simulation results are shown in Table 2:

[0116] Table 2: Algorithm hyperparameters

[0117] Figure 5 This figure shows the change in the total reward per episode during training using the improved SAC algorithm (hereinafter referred to as the ISAC algorithm). This curve was obtained by varying the random number seed and performing four Monte Carlo simulations. The solid line represents the mean, and the colored areas represent the variance. It is easy to see that the ISAC algorithm has good convergence speed and stability.

[0118] Figure 6 is the path planned by the algorithm in environment 1; Figure 7 is the rotation ratio of the rudder angle and the vector angle in environment 1; Figure 8 is the path planned by the algorithm in environment 2; Figure 9 is the rotation ratio of the rudder angle and the vector angle in environment 2; Figure 10 is the path planned by the algorithm in environment 3; Figure 11 is the rotation ratio of the rudder angle and the vector angle in environment 3. Figures 6 to 11The paths planned by the ISAC algorithm in different environments and the vector thruster deflection angles and rudder deflection angles required to generate the above paths are shown respectively. Among them, the blue dot represents the starting point of the AUV, the orange five-pointed star represents the target point that the AUV needs to reach, the yellow circle represents the obstacles in the environment, the blue solid line represents the path taken by the AUV, and the physical meaning of the deflection ratio represents the ratio of the actual rotation size of the deflection angle to the maximum deflection angle. It is not difficult to find that the algorithm can reach the target point in all three environments, and from the perspective of the rotation ratio of the rudder angle and the vector deflection angle, the proportion of the vector deflection angle used in all three environments is smaller than the proportion of the rudder angle used, thus avoiding the excessive use of the vector deflection angle that reduces the speed of the AUV, and at the same time improving the maneuverability of the AUV, allowing the AUV to reach the target point faster.

[0119] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and are intended to be encompassed by the claims of the present invention.

Claims

1. A vector-propelled AUV path planning method based on deep reinforcement learning, comprising: Step 1) Real-time acquisition of the AUV’s current state s based on sonar and its own inertial navigation information t , according to the parameterized strategy network π θ (a t |s t ) Output policy action a t , use the reward function to calculate the reward value r t , use the AUV dynamics equation to calculate the next state s t+1 , the tuple [s t ,a t ,r t ,s t+1 ]Deposit into experience pool Step 2) When the experience pool When the number of tuples in the Sample M tuples from ; go to step 3); Otherwise, go to step 1); Step 3) Under the maximum entropy reinforcement learning framework, the policy network parameters θ and the evaluation network parameters And the temperature parameter α is updated; Step 4) When the AUV meets the termination condition, the process ends; otherwise, go to step 1); The policy action a in step 1) t Modeled as follows: in, The proportional coefficient representing the rudder angle and the geometric deflection angle of the vector thruster is transformed as follows to obtain the actual output action: a t ←a t ·[d max ,x max ] Among them, δ max ,ξ max Represents the rudder angle limit and geometric deflection angle limit; Step 1) reward value r t for: r t ＝r z +r s +r d Among them, r z Represents the terminal reward, giving a larger positive feedback when reaching the target point and a smaller negative feedback when colliding with an obstacle; r s Represents immediate reward, r d Represents rewards related to rudder angle and vector geometry deflection.

2. The vector propulsion AUV path planning method based on deep reinforcement learning according to claim 1 is characterized in that: The step 1) also includes: Real-time acquisition of the AUV's current status based on sonar and its own inertial navigation information t 、Action a t , and the state s at the next moment t+1 ; According to the state s t 、Action a t , the current reward r is calculated by the reward function t ; The tuple [s t ,a t ,r t ,s t+1 ]Deposit into experience pool Continue the above steps until the experience pool The number of tuples in is greater than the set value N.

3. The vector propulsion AUV path planning method based on deep reinforcement learning according to claim 1 is characterized in that, The policy network and the evaluation network are two independent neural networks; the layers of each neural network are connected using batch normalization and bounded activation functions; The action distribution is modeled using Gaussian distribution, and the mean and standard deviation of the Gaussian distribution are finally output.

4. The vector propulsion AUV path planning method based on deep reinforcement learning according to claim 1 is characterized in that The step 3) comprises: Step 3-1) The loss function is used to evaluate the network Parameters Make updates; Step 3-2) J π (θ) is the loss function for the policy network π θ (a t |s t )’s parameters θ are updated; Step 3-3) Update the temperature parameter α using J(α) as the loss function.

5. The vector propulsion AUV path planning method based on deep reinforcement learning according to claim 4 is characterized in that: Step 3-1) loss function for: Where, |·| sg Its gradient is not required for characterization, represents the parameterized evaluation network, a t+1 Indicates the next action, s t+1 Indicates the state at the next moment, q t ,q t+1 Respectively represent when the input of the evaluation network is [s t ,a t ],[s t+1 ,a t+1 ], γ represents the discount factor, γ∈[0,1].

6. The vector propulsion AUV path planning method based on deep reinforcement learning according to claim 4 is characterized in that: Step 3-2) loss function J π (θ) is: Where D KL (·||·) is used to measure the difference between the two strategies, π θ (·||s t ) represents the state s t The strategy distribution under Representation in state s t The non-normalized probability distribution of the action-state value function under , Characterizes the normalization constant.

7. The vector-propelled AUV path planning method based on deep reinforcement learning according to claim 4 is characterized in that: The loss function J(α) in step 3-3) is: Where, Characterize a constant, Represents an expectation symbol, the action a inside the expectation symbol t Using parameterized strategy π θ Take samples.

8. The vector propulsion AUV path planning method based on deep reinforcement learning according to claim 4 is characterized in that: The termination condition of step 4) includes: the actual number of running scenes is greater than the maximum number of running scenes super parameter T me .

Images