An improved DQN-based path planning algorithm for unmanned ships

Abstract

The application discloses an unmanned ship path planning algorithm for improving DQN, relates to the technical field of unmanned ship path planning algorithms, and aims to solve the problems of poor detection capability and long training time of the DQN algorithm, and the technical scheme is as follows: S1: algorithm improvement: the Q-learning algorithm and the DQN algorithm are improved, and are substituted into a subsequently established model for calculation; S2: environment modeling; S3: reward function: the reward function of the unmanned ship path planning algorithm is improved; S4: DQN network design; S5: dynamic window algorithm design: a dynamic window method is used to calculate local path planning, the dynamic window method comprises three steps, namely, establishing a motion model, velocity sampling and trajectory evaluation; S6: DQN-DWA fusion algorithm: the DQN algorithm and the dynamic window method are fused, and the Q value in the DQN is used to guide the dynamic window method; and S7: simulation experiment. The effects of accurate path planning, sufficient local route planning and real-time accurate planning are achieved.

Description

Technical Field

[0001] This invention relates to the field of unmanned surface vessel (USV) path planning algorithm technology, and in particular to an USV path planning algorithm based on an improved DQN. Background Technology

[0002] With the continuous development of the economy and the increasing population, the amount of floating debris such as tree branches and leaves, crop weeds, plastic foam, and domestic garbage in reservoirs, rivers, lakes, and nearshore waters is increasing. Due to the influence of subjective and objective factors, the water surface is often covered with various floating objects, and the amount of floating debris on the water surface is increasing year by year, posing a threat to the safety of reservoirs, power stations, various water facilities, and drinking water. People are striving to adopt various methods to salvage floating debris, and unmanned cleaning boats are an important tool for identifying, picking up, and processing garbage on the water surface.

[0003] Unmanned surface vessel (USV) path planning technology requires USVs to autonomously plan a safe and economical optimal path connecting the starting point and the destination, which is another key technology for realizing intelligent decision-making of USV cleaning vessels.

[0004] However, in the actual aquatic environment, in addition to static obstacles such as reefs and buildings, there are also dynamic obstacles such as various ships. Therefore, unmanned surface vessel path planning technology not only needs to realize global path planning for known static obstacles, but also needs to realize local path planning based on unknown obstacles and dynamic obstacles. The existing DQN algorithm for detection has problems such as poor detection capability and long training time, which leads to many cases of damage to unmanned cleaning vessels. Summary of the Invention

[0005] The purpose of this invention is to provide an unmanned surface vessel path planning algorithm that can fully identify obstacles in water and perform path planning.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An unmanned surface vessel (USV) path planning algorithm based on improved Direct Quantity Detection (DQN) is proposed, and the algorithm is as follows:

[0008] S1: Algorithm Improvement: Improve the Q-learning algorithm and DQN algorithm, and substitute them into the subsequently established model for calculation;

[0009] S2: Environment Modeling: Use appropriate methods to establish an environment model for path planning. The environment modeling used is based on expanded circular and rectangular obstacles. The unmanned boat is regarded as a point mass occupying only one grid. After simplification, the state space of the unmanned boat is the two-dimensional coordinates of the unmanned boat in the map, while the motion space of the unmanned boat is four directions of motion.

[0010] S3: Reward Function: The reward function for the unmanned surface vessel path planning algorithm has been improved as follows:

[0011]

[0012] in, This is the distance from the unmanned vessel to the target. "round" represents the rounding function, which sets the reward function to a discrete value. Overall, the closer to the target point, the smaller the value, but not so small that it is different every two steps. At the same time, this value is negative. It is always negative if the destination is not reached, so that the agent will not get stuck in a local loop in order to avoid punishment.

[0013] S4: DQN Network Design: Using the leaky ReLU function, when x < 0, the coefficient 'a' ensures that there is still some output, preventing neurons from dying and no longer participating in parameter updates. The input of the neural network is the state space, i.e., the coordinates of the unmanned vessel, and the output is the action space, i.e., the four actions of the unmanned vessel: forward, backward, left, and right. It contains two neural networks: eval_net and target_net. eval_net stores the information of the current state, and target_net stores the information of the state at the next time step, thereby calculating the Q value. The two neural networks eval_net and target_net have the same structure. The input layer contains 32 neurons, and there are two hidden layers with 512 and 128 neurons respectively. The network structure can be visualized using the Tensorboard tool.

[0014] S5: Dynamic Window Algorithm Design: The dynamic window method is used to calculate local path planning. The dynamic window method consists of three steps: establishing a motion model, velocity sampling, and trajectory evaluation.

[0015] S6: DQN-DWA Fusion Algorithm: This algorithm combines the DQN algorithm with the dynamic window method, using the Q-value in DQN to guide the dynamic window method.

[0016] B1: Use RGB images as the simulation environment for unmanned surface vessel (USV) path planning. Read the grayscale values ​​of the map, set the black areas as passable areas in the USV map, and set the white areas as obstacles. Randomly initialize the starting point and ending point of the USV path planning for each time, while ensuring that the distance between the starting point and the ending point is not too close, and that it is not on the edge or inside of the obstacle, so as to achieve the effect of simulating dynamic obstacles.

[0017] B2: Establish the kinematic model of the unmanned surface vessel (USV), initialize its motion parameters, including velocity range, angular velocity range, maximum acceleration, maximum angular acceleration, and the resolution of velocity and rotational speed. Use a function to obtain the information of surrounding obstacles detected by the USV and their distances from the target point as state information. Then, update the state using the current parameters through a dynamic window method to obtain the next state and update the reward value. In the dynamic obstacle avoidance stage, the reward value is set as follows:

[0018]

[0019] Where CurDist represents the distance between the current position and the target point, i.e. the position after the dynamic window update, PreDist represents the distance between the position before the update and the target point. When the distance between the position and the target point becomes farther, a greater penalty will be imposed. τ is the distance threshold. dist(v,ω)<τ means that the risk of colliding with the obstacle at the current speed and angular velocity of the unmanned boat is very high, i.e., the line tracking is considered to have failed. dist(v,ω)>τ means that the current position is within the safe distance. When the destination has not been reached, the reward value is set to a negative value to avoid getting trapped in a local optimum.

[0020] B3: The algorithm updates the state by using a dynamic window method to update the state before and after the current state. The actions, reward values, and whether the target point has been reached are stored in the cache. When the cache is full, the algorithm learns once every 5 path planning iterations and updates the network weight parameters using gradient descent. This process is repeated until the destination is reached or an obstacle is encountered.

[0021] In this process, the greedy coefficient ε for selecting actions is set to a dynamic value. Each time an action needs to be selected during path planning, a random number σ∈[0,1] is generated. When σ is less than ε, an action is randomly selected; otherwise, the optimal action is selected. Let:

[0022] ε=ε·(0.99) episode

[0023] Here, episode represents the number of iterations, and ε is a value that gradually decreases with the number of iterations. When the number of iterations is relatively small, the random selection of actions may be more likely to avoid focusing too much on immediate benefits and getting trapped in a local optimum. When the number of iterations is sufficient, the Q value output by the network is more instructive and tends to select the optimal solution to reach the endpoint faster.

[0024] S7: Simulation Experiment: The improved DQN algorithm and Q-learning algorithm are applied simultaneously on a 15*10 map. The shortest path found by the two algorithms, as well as the step change graph and reward value graph during the training process, are statistically analyzed to determine the overall practical feasibility.

[0025] Furthermore, the Q-learning in S1 is improved as follows: Let Q(s,a) represent the reward that the agent can obtain by taking action a in state s, and construct a Q-table to store the value of Q based on the reward obtained by the agent according to different states and actions;

[0026] Let the state of the agent at step t be s. t The next state s t+1 Through the state transition equation T(a|s) t We obtain P(s) t+1 |s t ,a) indicates that the action a is performed to move from state s. t Transition to state s t+1 The probability, R(s) t+1 |s t ,a) means that the action a changes the state s. t Transition to state s t+1 The goal of the Q-learning algorithm is to find the policy expectation that maximizes the cumulative reward, which can be expressed by the formula:

[0027]

[0028] Where γ represents the attenuation coefficient, indicating that the greater the distance from the agent's current state, the smaller the impact;

[0029] The Q-learning algorithm uses the Bellman equation to solve for the optimal strategy of the Markov process. According to the Bellman equation, the state-value function in state Q(s,a) is defined as follows:

[0030] q π (s,a)=E π {r t+1 +γ[r t+2 +γ(r t+3 +...)]|A t =a,S t =s}

[0031] Where, r t+1 Let represent the reward value at time t+1, and γ represent the decay coefficient. The closer γ is to 1, the more attention is paid to the value of subsequent states; when γ is close to 0, it pays more attention to the impact of current benefits.

[0032] The function Q that yields the optimal value * (s,a)=max π Q π (s,a), that is:

[0033] Q * (s,a)=∑s' P(s'|s,a)[R(s'|s,a)+γmax a' Q * (s',a')]

[0034] The Q-learning algorithm uses the time difference method to update the Q-table, which combines dynamic programming and Monte Carlo sampling. The Q-table update formula is as follows:

[0035] Q(s t ,a)=Q(s t ,a)+α[r t +γmaxQ(s t+1 ,a)-Q(s t ,a)]

[0036] Where α∈(0,1) is the learning rate;

[0037] Traditional Q-learning algorithms typically employ a greedy algorithm to select actions. First, an exploration factor ε is determined. When selecting an action, the agent has an ε probability of choosing the optimal action and a 1-ε probability of randomly selecting an action, which can be represented as:

[0038]

[0039] Where σ∈[0,1] is a random variable, and rand(A) is a random action in the action set.

[0040] Furthermore, the DQN algorithm in S1 is designed as follows: the Q-learning algorithm uses a Q-Table to store Q values. However, when the environment faced by the agent and the actions to be taken are complex, using such a method will result in the amount of information contained in the two-dimensional table being too large, and it will take a lot of time to obtain each Q value.

[0041] The DQN algorithm uses a neural network to approximate the value function:

[0042]

[0043] In Q-learning algorithms, the predicted and target values ​​use the same Q-value. If the target value is also updated in the neural network, the network will not converge. This is because in supervised learning, the target value must be fixed. Therefore, DQN introduces a target Q-network to update the target value on top of the Q-network. The two network structures are the same, with the same initial weights; the only difference is that the Q-network is updated in each iteration, while the target Q-network is updated periodically.

[0044] The DQN algorithm uses gradient descent to update the neural network parameters. First, a loss function needs to be defined:

[0045]

[0046] in, Let ω be the weight parameters in the target Q network, and ω be the network parameters in the Q network. By differentiating these parameters, error backpropagation can be achieved.

[0047]

[0048] In reinforcement learning, the observed data is ordered, while in neural networks, the data is independent. Therefore, DQN addresses this problem by using experience replay, which involves storing experience data in real time using memory, thus making the data independent.

[0049] Furthermore, in S4, a motion model is established to model the unmanned vessel. WA is based on a realistic motion model, assuming the unmanned vessel cannot move omnidirectionally, but can only turn left, right, and forward. The distance traveled between adjacent time points is relatively short and can be considered a straight line. Therefore, the displacement increment during this period can be projected onto the X and Y axes of the world coordinate system. The distance traveled by the unmanned vessel over a period of time can be calculated by summing the displacement increments of each segment during that period.

[0050]

[0051] Where v is the linear velocity of the unmanned vessel, ω is the angular velocity of the unmanned vessel, and Δt is the minimum time interval for modeling the unmanned vessel.

[0052] Furthermore, the velocity sampling in S4 is constrained by the performance of the unmanned vessel itself and its motors, and there are upper and lower limits on the linear velocity v and angular velocity ω, which can be specifically expressed as:

[0053] V s ={(v,ω)|v∈[v min ,v max ],ω∈[ω min ,ω max ]}

[0054] On the other hand, due to limitations in motor torque performance, there are upper limits on the maximum acceleration and maximum angular acceleration. Under these limitations, the unmanned vessel can reach a certain speed V within a time interval Δt. d It can be represented as:

[0055]

[0056] in, This indicates the maximum acceleration and deceleration linear speed that the unmanned surface vessel can achieve. This indicates the maximum angular deceleration and acceleration that the unmanned surface vessel can achieve.

[0057] Furthermore, to prevent collisions caused by insufficient braking distance after the unmanned surface vessel detects obstacles, a certain safety distance should be maintained. Therefore, its speed range should meet the following requirements:

[0058]

[0059] Where dist(v,ω) is the shortest distance between the unmanned surface vessel and the obstacle.

[0060] Therefore, combining the three speed limits, the velocity space V of the unmanned ship at each moment is... R =V s ∩V d ∩V o .

[0061] Furthermore, in S4, after the trajectory evaluation meets the speed limit, several sets of trajectories that meet the requirements can be calculated using several sets of velocities in the speed space.

[0062]

[0063] Where, x t y t θ t dx, dy, and dθ represent the x-coordinate, y-coordinate, and orientation angle of the unmanned vessel at time t in the coordinate system. dx, dy, and dθ represent the changes of these three quantities over time dt. These three quantities can be obtained from velocity, acceleration, angular velocity, and angular acceleration, and therefore can be extrapolated from velocity space to trajectory space.

[0064] To select the optimal trajectory from the trajectory space, we introduce a trajectory evaluation function:

[0065] G(v,w)=αH(v,w)+βD(v,w)+γV(v,w)

[0066] Here, H(v,w) is the heading score, which is determined by the difference between the unmanned vessel's forward direction and the angle of its direction from the target point; D(v,w) is the distance score, which is determined by the distance of the unmanned vessel from the target point; and V(v,w) is the speed score, which mainly ensures that the unmanned vessel can reach the target point faster while ensuring safety.

[0067] In summary, the beneficial technical effects of the present invention are as follows:

[0068] 1. This paper proposes an improved DQN-based unmanned surface vessel (USV) path planning algorithm. To address the issue that the DQN algorithm is prone to getting trapped in local optima during global path planning, the reward function in the DQN algorithm is improved. To address the potential neuron death problem in the ReLU function of the DQN algorithm, the ReLU function is replaced with a leaky-ReLU function. Then, a global path planning algorithm for USVs based on the improved DQN is designed, achieving accurate path planning.

[0069] 2. This paper improves the DQN unmanned vessel path planning algorithm. Targeting the characteristics of local path planning, it combines the DWA algorithm with the DQN algorithm. The DQN algorithm is used to optimize the parameters of the evaluation function in the DWA algorithm. Then, a DQN-DWA local path planning algorithm is designed to achieve the effect of fully planning local routes.

[0070] 3. The path planning algorithm for unmanned vessels based on the improved DQN was tested through simulation experiments on the designed global and local path planning algorithms. The path planning algorithm based on the improved DQN has strong real-time performance and robustness, and achieves the effect of real-time and accurate planning. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the principle framework of the DQN algorithm of this invention;

[0072] Figure 2 This is a flowchart of the global path planning algorithm for the unmanned surface vessel of the present invention.

[0073] Figure 3 This is a flowchart of the local path planning algorithm for the unmanned surface vessel DQN-DWA of the present invention. Detailed Implementation

[0074] The method of the present invention will be further described in detail below with reference to the accompanying drawings.

[0075] Reference Figure 1 An unmanned surface vessel path planning algorithm based on improved DQN is proposed, and the algorithm is as follows:

[0076] S1: Algorithm Improvement: Improve the Q-learning algorithm and DQN algorithm, and substitute them into the subsequently established model for calculation;

[0077] Q-learning is a value-based reinforcement learning algorithm. Its algorithmic framework can be summarized by a Markov Decision Process (MDP). A Markov Decision Process includes elements such as environment, agent, action, and reward. Let Q(s,a) represent the reward that the agent can obtain by taking action a in state s. The reward obtained by the agent according to different states and actions is constructed into a Q-table to store the value of Q. This is the core idea of ​​Q-learning.

[0078] Let the state of the agent at step t be s. t The next state s t+1 Through the state transition equation T(a|s) t We obtain P(s) t+1 |s t ,a) indicates that the action a is performed to move from state s. t Transition to state s t+1 The probability, R(s) t+1 |s t ,a) means that the action a changes the state s. t Transition to state s t+1 The reward obtained. The goal of the Q-learning algorithm is to find the policy expectation that maximizes the cumulative reward, which can be expressed by the formula:

[0079]

[0080] Where γ represents the attenuation coefficient, indicating that the greater the distance from the agent's current state, the smaller the impact.

[0081] The Q-learning algorithm uses the Bellman equation to solve for the optimal strategy of the Markov process. According to the Bellman equation, the state-value function in state Q(s,a) is defined as follows:

[0082] q π (s,a)=E π {r t+1 +γ[r t+2 +γ(r t+3 +...)]|A t =a,S t =s}

[0083] Where, r t+1 The value represents the reward at time t+1, and γ represents the decay coefficient. The closer γ is to 1, the more attention is paid to the value of subsequent states; when γ is close to 0, it pays more attention to the impact of current benefits.

[0084] The function Q that yields the optimal value * (s,a)=max π Q π (s,a), that is:

[0085] Q * (s,a)=∑ s' P(s'|s,a)[R(s'|s,a)+γmax a' Q * (s',a')]

[0086] The Q-learning algorithm uses the time difference method to update the Q-table, which combines dynamic programming and Monte Carlo sampling. The Q-table update formula is as follows:

[0087] Q(s t ,a)=Q(s t ,a)+α[r t +γmaxQ(s t+1 ,a)-Q(s t ,a)]

[0088] Where α∈(0,1) is the learning rate.

[0089] Traditional Q-learning algorithms typically employ a greedy approach to select actions. First, an exploration factor ε is determined. When selecting an action, the agent has an ε-probability of choosing the optimal action and a 1-ε-probability of randomly selecting an action. This can be represented as:

[0090]

[0091] Where σ∈[0,1] is a random variable, and rand(A) is a random action in the action set;

[0092] DQN Algorithm Principle: Q-learning algorithms use Q-tables to store Q-values. However, when the environment and actions required by the agent are complex, this method results in the two-dimensional table requiring too much information, and calculating each Q-value takes a significant amount of time. Therefore, the DQN algorithm uses a neural network to approximate the value function.

[0093]

[0094] In the Q-learning algorithm, the predicted value and the target value use the same Q-value. If the target value is also updated in the neural network, the network will not converge. This is because in supervised learning, the target value must be fixed. Therefore, DQN introduces a target Q network to update the target value on top of the Q network. The two network structures are the same, and the initial weights are the same. The only difference is that the Q network is updated in each iteration, while the target Q network is updated at regular intervals.

[0095] The DQN algorithm uses gradient descent to update neural network parameters, which first requires defining a loss function:

[0096]

[0097] in, Let ω be the weight parameters in the target Q network, and ω be the network parameters in the Q network. By differentiating these parameters, error backpropagation can be achieved.

[0098]

[0099] In reinforcement learning, the observed data is ordered, while in neural networks, the data is independent. Therefore, DQN addresses this problem by using experience replay, which involves storing experience data in real time using memory, thus making the data independent.

[0100] S2: Environment Modeling: Use appropriate methods to establish an environment model for path planning. The environment modeling used is based on expanded circular and rectangular obstacles. The unmanned boat is regarded as a point mass occupying only one grid. After simplification, the state space of the unmanned boat is the two-dimensional coordinates of the unmanned boat in the map, while the motion space of the unmanned boat is four directions of motion.

[0101] In the global map, the volume of the unmanned boat can only occupy a very small part. Therefore, this paper regards the unmanned boat as a point mass occupying only one grid. After simplification, the state space of the unmanned boat is the two-dimensional coordinates of the unmanned boat in the map, and the motion space of the unmanned boat is four motion directions (w (forward), s (backward), a (left), d (right)).

[0102] S3: Reward Function: The reward function for the unmanned surface vessel path planning algorithm has been improved as follows:

[0103]

[0104] in, This is the distance from the unmanned vessel to the target. "round" represents the rounding function, which sets the reward function to a discrete value. Overall, the closer to the target point, the smaller the value, but not so small that it is different every two steps. At the same time, this value is negative. It is always negative if the destination is not reached, so that the agent will not get stuck in a local loop in order to avoid punishment.

[0105] Reference Figure 2 S4: DQN network design: Using the leaky ReLU function, when x < 0, the coefficient a ensures that there is still some output, preventing neurons from dying and no longer participating in parameter updates. The input of the neural network is the state space, i.e., the coordinates of the unmanned vessel, and the output is the action space, i.e., the four actions of the unmanned vessel: forward, backward, left, and right. It contains two neural networks: eval_net and target_net. eval_net stores the information of the current state, and target_net stores the information of the state at the next time step. The Q value is calculated in this way. The two neural networks eval_net and target_net have the same structure. The input layer contains 32 neurons, and there are two hidden layers with 512 and 128 neurons respectively. The network structure can be visualized using the Tensorboard tool.

[0106] S5: Dynamic Window Algorithm Design: The dynamic window method is used to calculate local path planning. The dynamic window method consists of three steps: establishing a motion model, velocity sampling, and trajectory evaluation.

[0107] A motion model is established to model the unmanned surface vessel (USV). WA is based on a realistic motion model, assuming the USV cannot move omnidirectionally, but can only turn left, right, and forward. The distance traveled between adjacent time points is relatively short and can be considered a straight line. Therefore, the displacement increment during this period can be projected onto the X and Y axes of the world coordinate system. The distance traveled by the USV over a period of time can be calculated by summing the displacement increments of each segment during that period.

[0108]

[0109] Where v is the linear velocity of the unmanned vessel, ω is the angular velocity of the unmanned vessel, and Δt is the minimum time interval for modeling the unmanned vessel.

[0110] In S4, velocity sampling is constrained by the performance of the unmanned vessel itself and its motors, with upper and lower limits on linear velocity v and angular velocity ω, which can be specifically expressed as:

[0111] V s ={(v,ω)|v∈[v min ,v max ],ω∈[ω min ,ω max ]}

[0112] On the other hand, due to limitations in motor torque performance, there are upper limits on the maximum acceleration and maximum angular acceleration. Under these limitations, the unmanned vessel can reach a certain speed V within a time interval Δt. d It can be represented as:

[0113]

[0114] in, This indicates the maximum acceleration and deceleration linear speed that the unmanned surface vessel can achieve. This indicates the maximum angular deceleration and acceleration that the unmanned surface vessel can achieve.

[0115] Furthermore, to prevent collisions caused by insufficient braking distance after the unmanned surface vessel detects obstacles, a certain safety distance should be maintained. Therefore, its speed range should meet the following requirements:

[0116]

[0117] Where dist(v,ω) is the shortest distance between the unmanned surface vessel and the obstacle.

[0118] Therefore, combining the three speed limits, the velocity space V of the unmanned ship at each moment is... R =V s ∩V d ∩V o .

[0119] In S4, trajectory evaluation involves calculating several sets of trajectories that meet the requirements after the speed limit is met, using several sets of speeds in the speed space.

[0120]

[0121] Where, x t y t θ t dx, dy, and dθ represent the x-coordinate, y-coordinate, and orientation angle of the unmanned vessel at time t in the coordinate system. dx, dy, and dθ represent the changes of these three quantities over time dt. These three quantities can be obtained from velocity, acceleration, angular velocity, and angular acceleration, and therefore can be extrapolated from velocity space to trajectory space.

[0122] To select the optimal trajectory from the trajectory space, we introduce a trajectory evaluation function:

[0123] G(v,w)=αH(v,w)+βD(v,w)+γV(v,w)

[0124] Here, H(v,w) is the heading score, determined by the difference between the unmanned vessel's forward direction and its angular orientation relative to the target point; D(v,w) is the distance score, determined by the distance of the unmanned vessel from the target point; and V(v,w) is the speed score, which primarily ensures that the unmanned vessel reaches the target point faster while maintaining safety.

[0125] Reference Figure 3 S6: DQN-DWA fusion algorithm: The DQN algorithm is fused with the dynamic window method, and the Q value in DQN is used to guide the dynamic window method;

[0126] B1: Use RGB images as the simulation environment for unmanned surface vessel (USV) path planning. Read the grayscale values ​​of the map, set the black areas as passable areas in the USV map, and set the white areas as obstacles. Randomly initialize the starting point and ending point of the USV path planning for each time, while ensuring that the distance between the starting point and the ending point is not too close, and that it is not on the edge or inside of the obstacle, so as to achieve the effect of simulating dynamic obstacles.

[0127] B2: Establish the kinematic model of the unmanned surface vessel (USV), initialize its motion parameters, including velocity range, angular velocity range, maximum acceleration, maximum angular acceleration, and the resolution of velocity and rotational speed. Use a function to obtain the information of surrounding obstacles detected by the USV and their distances from the target point as state information. Then, update the state using the current parameters through a dynamic window method to obtain the next state and update the reward value. In the dynamic obstacle avoidance stage, the reward value is set as follows:

[0128]

[0129] Where CurDist represents the distance between the current position and the target point, i.e. the position after the dynamic window update, PreDist represents the distance between the position before the update and the target point. When the distance between the position and the target point becomes farther, a greater penalty will be imposed. τ is the distance threshold. dist(v,ω)<τ means that the risk of colliding with the obstacle at the current speed and angular velocity of the unmanned boat is very high, i.e., the line tracking is considered to have failed. dist(v,ω)>τ means that the current position is within the safe distance. When the destination has not been reached, the reward value is set to a negative value to avoid getting trapped in a local optimum.

[0130] B3: The algorithm updates the state by using a dynamic window method to update the state before and after the current state. The actions, reward values, and whether the target point has been reached are stored in the cache. When the cache is full, the algorithm learns once every 5 path planning iterations and updates the network weight parameters using gradient descent. This process is repeated until the destination is reached or an obstacle is encountered.

[0131] In this process, the greedy coefficient ε for selecting actions is set to a dynamic value. Each time an action needs to be selected during path planning, a random number σ∈[0,1] is generated. When σ is less than ε, an action is randomly selected; otherwise, the optimal action is selected. Let:

[0132] ε=ε·(0.99) episode

[0133] Here, episode represents the number of iterations, and ε is a value that gradually decreases with the number of iterations. When the number of iterations is relatively small, the random selection of actions may be more likely to avoid focusing too much on immediate benefits and getting trapped in a local optimum. When the number of iterations is sufficient, the Q value output by the network is more instructive and tends to select the optimal solution to reach the endpoint faster.

[0134] S7: Simulation Experiment: The improved DQN algorithm and Q-learning algorithm are applied simultaneously on a 15*10 map. The shortest path found by the two algorithms, as well as the step change graph and reward value graph during the training process, are statistically analyzed to determine the overall practical feasibility.

[0135] In deep reinforcement learning, the selection of algorithm parameters not only directly determines whether a network model capable of completing the task can be trained, but is also crucial for improving the training speed of the algorithm. The training parameters of the DQN network in the global path planning simulation experiment in this paper are as follows:

[0136] Table 4.1 Training parameters of the DQN network for global path planning of unmanned surface vessels

[0137]

[0138] This paper quantitatively compares the shortest number of steps and running time of the two algorithms. It reveals that when the map is relatively simple, both Q-learning and the improved DQN algorithm can find the optimal path. However, when the map is more complex, Q-learning fails to find the optimal path within a limited number of training iterations, while the improved DQN algorithm still finds the optimal path. More importantly, the improved DQN algorithm consistently outperforms the Q-learning algorithm in running time, especially with more complex maps. This is because the length of the Q-Table increases, requiring a large amount of information to be stored in a two-dimensional table. Calculating the Q-value using a neural network avoids this problem. The algorithm used in this paper reduces the running time by 3.3%, 56.5%, and 65.6% compared to the Q-learning algorithm in simulation experiments on three maps, respectively. Simulation experiments demonstrate that the algorithm used in this paper has practical applicability in static planning for unmanned vessels.

[0139] Table 4.2 Comparison of Q-learning and improved DQN algorithms

[0140]

[0141] This paper uses an RGB image composed of black and white pixels as a simulation environment map for local path planning of unmanned ships. The map size is 800*800. The starting point and ending point are randomly generated for each path planning. Therefore, the environment is not prior knowledge for the unmanned ship. It needs to explore obstacles during the journey, which falls under the category of local planning.

[0142] Table 4.3 Training parameters of the DQN-DWA algorithm for local path planning of unmanned surface vessels

[0143]

[0144]

[0145] It can be observed that the average Qmax value rises rapidly at the beginning of the iteration. In the later stages of the iteration, the Qmax value changes within a certain range but still fluctuates. This indicates that in the later stages of the iteration, the deep reinforcement network is still learning and updating, but the output value only changes slightly within a certain range. Observing the R value, it can be seen that the R value tends to stabilize after 150 iterations. This indicates that the unmanned ship can achieve a stable success rate in path planning. Only when the ratio of failures to successes is relatively stable can the average reward value be basically the same. It can be intuitively seen that the success rate converges around 0.999 after about 140 iterations, proving that the DQN-DWA fusion algorithm has strong robustness. Before the fusion with the DQN algorithm, although the success rate of the DWA algorithm is unstable in the face of dynamic and complex environments, it can still reach 0.993 when the success rate is low. This shows that as long as the parameters are reasonable, the DWA algorithm itself already has a high success rate. The role of the DQN algorithm is mainly to learn and change the three coefficients in the DWA evaluation function, making the DWA algorithm more robust in dynamic environments.

[0146] The embodiments described herein are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An unmanned surface vessel path planning algorithm based on improved DQN, characterized in that: The algorithm is as follows: S1: Algorithm Improvement: Improve the Q-learning algorithm and DQN algorithm, and substitute them into the subsequently established model for calculation; S2: Environment Modeling: Use appropriate methods to establish an environment model for path planning. The environment modeling used is based on expanded circular and rectangular obstacles. The unmanned boat is regarded as a point mass occupying only one grid. After simplification, the state space of the unmanned boat is the two-dimensional coordinates of the unmanned boat in the map, while the motion space of the unmanned boat is four directions of motion. S3: Reward Function: The reward function for the unmanned surface vessel path planning algorithm has been improved as follows: in, This is the distance from the unmanned vessel to the target. "round" represents the rounding function, which sets the reward function to a discrete value. Overall, the closer to the target point, the smaller the value, but not so small that it is different every two steps. At the same time, this value is negative. It is always negative if the destination is not reached, so that the agent will not get stuck in a local loop in order to avoid punishment. S4: DQN Network Design: Using the leaky ReLU function, when x < 0, the coefficient 'a' ensures that there is still some output, preventing neurons from dying and no longer participating in parameter updates. The input of the neural network is the state space, i.e., the coordinates of the unmanned vessel, and the output is the action space, i.e., the four actions of the unmanned vessel: forward, backward, left, and right. It contains two neural networks: eval_net and target_net. eval_net stores the information of the current state, and target_net stores the information of the state at the next time step, thereby calculating the Q value. The two neural networks eval_net and target_net have the same structure. The input layer contains 32 neurons, and there are two hidden layers with 512 and 128 neurons respectively. The network structure can be visualized using the Tensorboard tool. S5: Dynamic Window Algorithm Design: The dynamic window method is used to calculate local path planning. The dynamic window method consists of three steps: establishing a motion model, velocity sampling, and trajectory evaluation. S6: DQN-DWA Fusion Algorithm: This algorithm combines the DQN algorithm with the dynamic window method, using the Q-value in DQN to guide the dynamic window method. B1: Use RGB images as the simulation environment for unmanned surface vessel (USV) path planning. Read the grayscale values ​​of the map, set the black areas as passable areas in the USV map, and set the white areas as obstacles. Randomly initialize the starting point and ending point of the USV path planning for each time, while ensuring that the distance between the starting point and the ending point is not too close, and that it is not on the edge or inside of the obstacle, so as to achieve the effect of simulating dynamic obstacles. B2: Establish the kinematic model of the unmanned surface vessel (USV), initialize its motion parameters, including velocity range, angular velocity range, maximum acceleration, maximum angular acceleration, and the resolution of velocity and rotational speed. Use a function to obtain the information of surrounding obstacles detected by the USV and their distances from the target point as state information. Then, update the state using the current parameters through a dynamic window method to obtain the next state and update the reward value. In the dynamic obstacle avoidance stage, the reward value is set as follows: Where CurDist represents the distance between the current position and the target point, i.e. the position after the dynamic window update, PreDist represents the distance between the position before the update and the target point. When the distance between the position and the target point becomes farther, a greater penalty will be imposed. τ is the distance threshold. dist(v,ω)<τ means that the risk of colliding with the obstacle at the current speed and angular velocity of the unmanned boat is very high, i.e., the line tracking is considered to have failed. dist(v,ω)>τ means that the current position is within the safe distance. When the destination has not been reached, the reward value is set to a negative value to avoid getting trapped in a local optimum. B3: The algorithm updates the state by using a dynamic window method to update the state before and after the current state. The actions, reward values, and whether the target point has been reached are stored in the cache. When the cache is full, the algorithm learns once every 5 path planning iterations and updates the network weight parameters using gradient descent. This process is repeated until the destination is reached or an obstacle is encountered. In this process, the greedy coefficient ε for selecting actions is set to a dynamic value. Each time an action needs to be selected during path planning, a random number σ∈[0,1] is generated. When σ is less than ε, an action is randomly selected; otherwise, the optimal action is selected. Let: e=e·(0.99) episode Here, episode represents the number of iterations, and ε is a value that gradually decreases with the number of iterations. When the number of iterations is relatively small, the random selection of actions may be more likely to avoid focusing too much on immediate benefits and getting trapped in a local optimum. When the number of iterations is sufficient, the Q value output by the network is more instructive and tends to select the optimal solution to reach the endpoint faster. S7: Simulation Experiment: The improved DQN algorithm and Q-learning algorithm are applied simultaneously on a 15*10 map. The shortest path found by the two algorithms, as well as the step change graph and reward value graph during the training process, are statistically analyzed to determine the overall practical feasibility.

2. The unmanned surface vessel path planning algorithm based on improved DQN according to claim 1, characterized in that: The Q-learning improvement in S1 is as follows: Let Q(s,a) represent the reward that the agent can obtain by taking action a in state s, and construct a Q-table to store the value of Q based on the reward obtained by the agent according to different states and actions; Let the state of the agent at step t be s. t The next state s t+1 Through the state transition equation T(a|s) t We obtain P(s) t+1 |s t ,a) indicates that the action a is performed to change state s. t Transition to state s t+1 The probability, R(s) t+1 |s t ,a) means that the action a changes the state s. t Transition to state s t+1 The goal of the Q-learning algorithm is to find the policy expectation that maximizes the cumulative reward, which can be expressed by the formula: Where γ represents the attenuation coefficient, indicating that the greater the distance from the agent's current state, the smaller the impact; The Q-learning algorithm uses the Bellman equation to solve for the optimal strategy of the Markov process. According to the Bellman equation, the state-value function in state Q(s,a) is defined as follows: q π (s,a)=E π {r t+1 +γ[r t+2 +γ(r t+3 +...)]|A t =a,S t =s} Where, r t+1 Let represent the reward value at time t+1, and γ represent the decay coefficient. The closer γ is to 1, the more attention is paid to the value of subsequent states; when γ is close to 0, it pays more attention to the impact of current benefits. The function Q that yields the optimal value * (s,a)=max π Q π (s,a), that is: Q * (s,a)=∑ s' P(s'|s,a)[R(s'|s,a)+γmax a' Q * (s',a')] The Q-learning algorithm uses the time difference method to update the Q-table, which combines dynamic programming and Monte Carlo sampling. The Q-table update formula is as follows: Q(s t ,a)=Q(s t ,a)+α[r t +γmaxQ(s t+1 ,a)-Q(s t ,a)] Where α∈(0,1) is the learning rate; Traditional Q-learning algorithms typically employ a greedy algorithm to select actions. First, an exploration factor ε is determined. When selecting an action, the agent has an ε probability of choosing the optimal action and a 1-ε probability of randomly selecting an action, which can be represented as: Where σ∈[0,1] is a random variable, and rand(A) is a random action in the action set.

3. The unmanned surface vessel path planning algorithm based on improved DQN according to claim 2, characterized in that: The DQN algorithm in S1 is designed as follows: The Q-learning algorithm uses a Q-Table to store Q values. However, when the environment faced by the agent and the actions to be taken are complex, using such a method will result in the amount of information required by the two-dimensional table being too large, and it will take a lot of time to obtain each Q value. The DQN algorithm uses a neural network to approximate the value function: In Q-learning algorithms, the predicted and target values ​​use the same Q-value. If the target value is also updated in the neural network, the network will not converge. This is because in supervised learning, the target value must be fixed. DQN introduces a target Q-network to update the target value, building upon the Q-network. The two networks have the same structure and initial weights; the only difference is that the Q-network is updated in each iteration, while the target Q-network is updated periodically. The DQN algorithm uses gradient descent to update the neural network parameters. First, a loss function needs to be defined: in, Let ω be the weight parameters in the target Q network, and ω be the network parameters in the Q network. By differentiating these parameters, error backpropagation can be achieved. In reinforcement learning, the observed data is ordered, while in neural networks, the data is independent. Therefore, DQN uses experience replay to solve this problem, that is, by using memory to store experience data in real time, thereby making the data independent.

4. The unmanned surface vessel path planning algorithm based on improved DQN according to claim 3, characterized in that: The motion model established in S4 is used to model the unmanned vessel. WA is based on a realistic motion model, assuming the unmanned vessel cannot move omnidirectionally, but can only turn left, right, and forward. The distance traveled between adjacent time points is relatively short and can be considered a straight line. Therefore, the displacement increment during this period can be projected onto the X and Y axes of the world coordinate system. The distance traveled by the unmanned vessel over a period of time can be calculated by summing the displacement increments of each segment during that period. Where v is the linear velocity of the unmanned vessel, ω is the angular velocity of the unmanned vessel, and Δt is the minimum time interval for modeling the unmanned vessel.

5. The unmanned surface vessel path planning algorithm based on improved DQN according to claim 4, characterized in that: The velocity sampling in S4 is constrained by the performance of the unmanned vessel itself and its motors, and there are upper and lower limits on the linear velocity v and angular velocity ω, which can be specifically expressed as: V s ={(v,ω)|v∈[v min ,v max ],ω∈[ω min ,ω max ]} On the other hand, due to limitations in motor torque performance, there are upper limits on the maximum acceleration and maximum angular acceleration. Under these limitations, the unmanned vessel can reach a certain speed V within a time interval Δt. d It can be represented as: in, This indicates the maximum acceleration and deceleration linear speed that the unmanned surface vessel can achieve. This indicates the maximum angular deceleration and acceleration that the unmanned surface vessel can achieve. In addition, to prevent collisions caused by insufficient braking distance after the unmanned surface vessel detects an obstacle, a certain safety distance should be maintained. Therefore, its speed range should meet the following requirements: Where dist(v,ω) is the shortest distance between the unmanned surface vessel and the obstacle. Therefore, combining the three speed limits, the velocity space V of the unmanned ship at each moment is... R =V s ∩V d ∩V o .

6. The unmanned surface vessel path planning algorithm based on improved DQN according to claim 5, characterized in that: In S4, trajectory evaluation involves calculating several sets of trajectories that meet the requirements after the speed limit is met, based on several sets of speeds in the speed space. Where, x t y t θ t Let dx, dy, and dθ represent the x-coordinate, y-coordinate, and orientation angle of the unmanned vessel at time t in the coordinate system. dx, dy, and dθ represent the changes of these three quantities over time dt. These three quantities can be obtained from velocity, acceleration, angular velocity, and angular acceleration, thus allowing calculations from velocity space to trajectory space. To select the optimal trajectory from the trajectory space, we introduce a trajectory evaluation function: G(v,w)=αH(v,w)+βD(v,w)+γV(v,w) Here, H(v,w) is the heading score, which is determined by the difference between the unmanned vessel's forward direction and the angle of its direction from the target point; D(v,w) is the distance score, which is determined by the distance of the unmanned vessel from the target point; and V(v,w) is the speed score, which mainly ensures that the unmanned vessel can reach the target point faster while ensuring safety.

Images