Ship path planning method based on deep reinforcement learning

Abstract

The invention discloses a ship path planning method based on deep reinforcement learning. The ship path planning method is suitable for a complex sea area environment. Through environment modeling and dynamic obstacle identification, a detailed state space is constructed, and perception of the environment is enhanced by using an artificial potential field. An ID3QN-PER algorithm is adopted to train a model, an adaptive exploration strategy and a network parameter updating mechanism are designed, and navigation safety and efficiency in a dynamic environment are ensured. The method further comprises the steps of discretizing an action space, designing an obstacle avoidance strategy in combination with a COLREGS specification, smoothing a route through a B-spline technology, adjusting a navigation strategy according to real-time environment data, and improving the real-time performance and the global optimization performance of route planning.

Description

Technical Field

[0001] The present invention belongs to the fields of ship transportation and information technology, and particularly relates to a ship path planning method based on deep reinforcement learning. Background Art

[0002] In the fields of modern shipping and ocean engineering, ship path planning has always been a crucial research topic. Ship path planning refers to determining the optimal or sub-optimal route for a ship to sail from the starting point to the target point under given goals and constraints. Compared with the land traffic environment, the sea area environment is more complex. Due to the influence of various factors such as geographical environment, meteorological conditions, the ship's own performance, and the dynamics of other ships during sea navigation, traditional path planning methods often struggle to balance the real-time nature and global optimality of navigation simultaneously. In recent years, new-generation intelligent algorithms such as deep learning and reinforcement learning have been introduced into the path planning field, forming a new research direction of deep reinforcement learning (DRL), which provides new solutions and technical means for ship path planning.

[0003] In traditional path planning methods, common ones include search-based algorithms and planning methods based on potential fields, graph theory methods, or optimal control theory. These methods can usually obtain good path solutions in a static and known environment. However, in the face of common dynamic and uncertain sea environments, such as the occurrence of wind waves, tides, and other moving ships, or when it is necessary to meet constraints such as the International Regulations for Preventing Collisions at Sea (COLREGS), these methods often have deficiencies in terms of real-time performance, global search efficiency, and comprehensive consideration of the actual navigation characteristics of ships. In addition, for problems with high-dimensional state spaces and long-time decision sequences, traditional planning methods usually need to perform large-scale traversals in the state space or search tree, consuming a significant amount of computing time and making it difficult to adapt to the real-time and complex application scenarios in the offshore area.

[0004] With the development of computer vision, deep learning, and reinforcement learning, researchers have gradually applied deep reinforcement learning to the path planning field. Deep reinforcement learning combines the powerful feature extraction and representation capabilities of deep learning and the outstanding advantages of reinforcement learning in sequential decision-making problems, and can obtain better decision-making strategies by continuously interacting with the environment in a complex high-dimensional environment. Since a ship needs to consider multiple factors such as its own heading, speed, dynamic characteristics, and obstacle distribution during navigation, and these factors change significantly under different water areas or meteorological conditions, deep reinforcement learning can provide powerful adaptive and perception capabilities for intelligent ship path planning.

[0005] At present, many literatures and patents have begun to focus on the research idea of "deep reinforcement learning + path planning". Patent No. CN202410519691.X proposes a method for dynamic path planning of unmanned aerial vehicles based on an improved D3QN network. This patent solution also adopts the combination of Double DQN and Dueling DQN, aiming to improve the route planning effect of unmanned aerial vehicles in complex airspace environments. Similar to the present invention, it focuses on improving the learning stability of the network, reducing the overestimation risk, and introducing an adaptive greedy strategy. The differences are as follows: It targets the unmanned aerial vehicle environment, and the flight path constraints are quite different from the maritime collision avoidance rules. At the same time, in terms of the design of the reward function and the construction of the environmental state space, it mainly faces aerial obstacles and meteorological factors. The present invention focuses on ship path planning in the marine environment, introduces artificial potential fields and COLREGS rules constraints to meet maritime navigation requirements, and further conducts dynamic interactions with other moving vessels.

[0006] Patent No. CN202411013656.7 proposes a path planning method for a bionic cleaning robot in public restrooms based on deep learning. This patent focuses on the problem of the moving path planning of indoor bionic cleaning robots. Although it adopts the technical framework of reinforcement learning and deep learning, the types of obstacles and the planning scenarios are quite different from those of maritime navigation, and it does not involve maritime collision avoidance rules and ship motion characteristics at all. The present invention specifically aims at dynamic obstacle avoidance of ships in complex offshore environments, and emphasizes the adaptation to maritime requirements such as nautical chart environments, tidal changes, and meeting regulations.

[0007] Patent No. CN202211014749.2 proposes a method and system for mapless path planning of robots based on deep reinforcement learning. This patent also uses deep reinforcement learning in its core principle to conduct robot path planning, but its main innovation focuses on visual recognition and obstacle avoidance decision-making in mapless scenarios. Compared with the present invention, it not only considers static geographical information (such as reefs and islands), but also takes into account other moving ships and dynamic factors such as tides and winds and waves, and sets up an artificial potential field enhancement and B-spline smoothing mechanism to better meet the route requirements of ships.

[0008] However, when directly applying the DQN (Deep Q-Network) algorithm to ship path planning, the following main problems will still be encountered:

[0009] The contradiction between the complexity of the state space and the learning efficiency. In the offshore navigation environment, ships may face various static and dynamic obstacles, such as reefs, shipwrecks, other moving ships, and even marine organisms, which will generate an extremely complex and high-dimensional state space. If only the most basic DQN is used for characterization and decision-making, the training process of the Q-value function will become extremely slow and unstable.

[0010] Path redundancy and non-smoothness. Traditional DQN only focuses on immediate rewards during the update process. Once there is a lack of reasonable modeling of route smoothness or turning costs, the planned paths output often have a large number of redundant points or cannot maintain a relatively smooth sailing trajectory. For targets such as ships with large inertia and high momentum in water areas, frequent large-angle turns or excessive avoidance will increase fuel consumption and reduce sailing safety.

[0011] Real-time collision avoidance in a dynamic environment. The marine environment is significantly different from the scenarios of indoor robots or land vehicles. Various environmental variables and uncertain factors are more prominent, and it is also restricted by international maritime collision avoidance rules. If the algorithm does not have special constraints or designs for COLREGS, it is prone to making action selections that do not meet the collision avoidance requirements when dealing with complex encounter scenarios. Summary of the Invention

[0012] Based on the above technical background and the deficiencies of the existing technology, the present invention proposes a ship path planning method based on deep reinforcement learning, which combines an improved ID3QN (simultaneously integrating Double DQN and Dueling DQN) and a prioritized experience replay (PER) reinforcement learning structure, adds artificial potential field forces to the state space, explicitly reflects compliance with maritime collision avoidance rules in the reward function, and finally corrects the track through B-spline curves, thereby ensuring both the safety and effectiveness of path planning and providing a trajectory output with higher feasibility and better smoothness for the ship control system, and solving the problems that the existing technology is difficult to balance complex environment perception, efficient obstacle avoidance, and maritime specification constraints. A ship path planning method based on deep reinforcement learning proposed by the present invention includes the following steps:

[0013] S1: Environmental modeling and encounter situation recognition, specifically including the following steps:

[0014] S1-1: Discretize and model the target sea area to obtain a feasible region; at each discrete time t, let the current position of the ship be (x t , y t ), and define the next movable position of the ship according to Equation (1);

[0015] P t+1 = {(x, y)|(x, y) ∈ G f , x ∈ [x t - 1, x t + 1], y ∈ [y t - 1, y t + 1]} (1)

[0016] Wherein, G f is the ship's feasible region; x t , y tThey are the abscissa and ordinate of the current position of the ship respectively;

[0017] S1-2: For static obstacles such as reefs and shipwrecks and other moving ship dynamic obstacles existing in the sea environment, monitor and update the distribution information of the obstacles in the feasible area in real time;

[0018] S1-3: Divide the encounter situations of ships in the sea area according to the COLREGS regulations, and design corresponding obstacle avoidance strategies according to the relative course and speed relationship;

[0019] S2: State space construction and artificial potential field enhancement, which specifically include the following steps:

[0020] S2-1: Compose the key information that can be obtained by the ship in the sea environment into the state space S, including the current position of the ship (x t , y t ), the distance between the ship and the target point, the current course angle of the ship, and the relative positions of other dynamic ships in the environment;

[0021] S2-2: Introduce an artificial potential field to strengthen the description of the environment;

[0022] S2-3: The final state space S satisfies Equation (2), ensuring a comprehensive description of the dynamic changes of the ship itself and the surrounding environment, where Equation (2) is:

[0023] S = {x ship , y ship , d se , step ship , θ ship , f x , f y , x obs , y obs} (2)

[0024] Among them, x ship and y ship represent the current position of the ship; d se represents the distance between the ship and the end point; step ship represents the current number of steps of the ship; θ ship represents the current angle of the ship; f x and f y respectively represent the potential field forces received by the ship on the x-axis and y-axis currently; x obs and y obs represent the positions of the current dynamic obstacles;

[0025] S3: Action space design. Discretize the ship's course adjustment operation to obtain the action set A, which satisfies Equation (3). Each discrete action represents the change in the steering angle relative to the current course, including keeping the course unchanged, small-angle steering, medium-angle steering, and large-angle steering. Equation (3) is as follows:

[0026] A = {0°, 15°, -15°, 30°, -30°, t5°, -t5°, 60°, -60°} (3)

[0027] Among them, 0° means keeping the current course unchanged; 15° and -15° mean slightly adjusting the ship's direction, which is suitable for fine-tuning the course; 30° and -30° mean medium-angle steering adjustment, which is suitable for avoiding nearby obstacles and can achieve obvious direction correction; 60° and -60° mean large-angle course adjustment, which is used to cope with emergency obstacle avoidance or large-angle steering requirements.

[0028] S4: Comprehensive reward function design. Define the total reward function R as shown in Equation (4), and the specific calculation formula is:

[0029] R t = R τ1 + R τ2 + R τ3 + R τt + R τ5 (4)

[0030] Among them, R τ1 is the distance reward function; R τ2 is the steering reward function; R τ3 is the obstacle and boundary reward function; R τt is the end point reward function; R τ5 is the dynamic collision avoidance reward function. To make the ship consider both safety and efficiency during path planning, it specifically includes the following characteristics:

[0031] The distance reward function judges whether the ship is approaching the target area according to the change in the distance between the ship and the target point. If the ship is closer to the target, the positive reward is greater; if the distance increases, the reward decreases. The calculation formula is:

[0032] R τ1 = k1·(d t - d t+1 ) (5)

[0033] Among them, k1 is the weight coefficient; d t is the distance between the ship and the target point at time t; d t+1 is the distance between the ship and the target point at time t + 1;

[0034] The turning reward imposes a moderate negative penalty on frequent or large-angle unnecessary turns, guiding the ship to maintain a stable course as much as possible under feasible conditions; the calculation formula is:

[0035] R τ2 = k2·cos(3*θ t ) (6)

[0036] where k2 is the weight coefficient; θ t indicates the turning angle of the ship's movement at the next moment;

[0037] The obstacle and boundary reward gives a negative reward when the ship approaches an obstacle or exceeds the environmental boundary, to restrict the safe navigation of the ship; the calculation formula is:

[0038]

[0039] where k3 is a negative constant; d o is the minimum distance between the ship and the obstacle or boundary, and d s is the actual distance between the ship and the obstacle or boundary;

[0040] The end point reward gives an additional high positive reward when the ship successfully reaches the target position, to highlight the priority of task completion; the calculation formula is:

[0041] R τt = k t (8)

[0042] where k t is a positive constant;

[0043] The dynamic collision avoidance reward determines the encounter situation based on information such as the relative position, speed, and course between the own ship and the target ship; the calculation formula is:

[0044]

[0045] where r ho is the reward for the head-on encounter situation; r ot is the reward for the overtaking situation; r cleft is the reward for the port side crossing; r cright is the reward for the starboard side crossing; a ta is the action to be taken by the own ship in the next step; θ own is the angle between the direction of the own ship's travel and the x-axis direction; θ tar is the angle between the direction of the target ship's travel and the x-axis direction;

[0046] S5: Training based on the ID3QN-PER deep reinforcement learning, specifically including the following structural features:

[0047] S5-1: Use a dual-network structure that uses two neural networks with the same structure but independent parameters as the current Q-network and the target Q-network respectively, and calculate the target Q-value through Equation (10):

[0048]

[0049] Among them, represents the target network; argmax a′ Q(s′, a′, θ) is the optimal action generated by the current network; r is the immediate reward; γ is the discount factor;

[0050] S5-2: Use a dueling network structure that splits the Q-value into a state value function V(s) and an advantage function A(s, a), and combines them through Equation (11):

[0051]

[0052] Among them, V(s) is the value of state s; A(s, a) is the relative advantage of action a in state s; |A| is the number of actions; is the average advantage value of all actions;

[0053] S5-3: Prioritized experience replay, prioritize and sample historical experiences based on TD error to improve the utilization efficiency of key samples;

[0054] S5-4: Adaptive exploration strategy, balance exploration and exploitation through non-linearly decaying ε-greedy method, expand the search scope in the early stage of training, and gradually utilize existing experiences in the later stage;

[0055] S5-5: Network parameter update, periodically sample a small batch of data from the experience pool for gradient descent, and synchronize the current Q-network parameters to the target Q-network until a convergent policy model is obtained;

[0056] S6: Path generation and B-spline smoothing, specifically including the following steps:

[0057] S6-1: After the ID3QN-PER algorithm is trained, the optimal action a can be quickly inferred from the trained Q-network according to the current state S during actual navigation to achieve real-time obstacle avoidance and navigation;

[0058] S6-2: For the discrete navigation path output by the algorithm, extract the turning points as the control points of the B-spline curve. Among them, the calculation formula of the B-spline curve is:

[0059]

[0060] Among them, C(u) is the point on the curve; n is the number of control points; B iis the control point; u is the node vector, which controls the shape of the basis function and the segmentation of the curve; N i,k (u) represents the k-order B-spline basis function;

[0061] S6-3: The control points are interpolated into a continuous curve, and the B-spline basis function is recursively defined as a piecewise polynomial by combining equations (13) and (14) to achieve smooth path generation:

[0062]

[0063] Among them, x i is the element in the node vector;

[0064] S6-4: Evaluate the B-spline smoothed route. If there is a conflict with a dynamic obstacle or environmental change, return to step S5 to retrain and update the strategy.

[0065] S7: Navigation execution and adaptive update, specifically including the following steps:

[0066] S7-1: The smoothed path is used as the final route for the ship to navigate using the ship's own steering and speed control systems.

[0067] S7-2: Monitor sea conditions in real time during navigation. Once a new ship is detected or the obstacle situation changes significantly, repeat steps S2 to S6 above and adjust the strategy and path again in combination with the ID3QN-PER algorithm to ensure that COLREGS specifications are always met in a dynamic environment and to improve navigation efficiency and safety.

[0068] As a technical preferred solution of the present invention, the COLREGS specification described in step S1-3 divides the encounter situations into head-on encounter situations, overtaking situations and crossing encounter situations, and further subdivides them into port crossing and starboard crossing situations. The corresponding obstacle avoidance strategy is designed according to the relative heading and speed relationship; among them, when the overtaking scenario is determined to be a head-on encounter scenario, the own ship should actively make an evasive move. In the case of a head-on encounter scenario, both parties will simultaneously travel to the starboard direction, while in the crossing scenario, the ship on the starboard direction will have priority to pass, thereby standardizing the heading adjustment principles in different situations.

[0069] As a preferred technical solution of the present invention, step S2-2 introduces an artificial potential field to enhance the characterization of the environment, specifically comprising the following steps:

[0070] A1: Use formula (15) to define the target attraction potential field function U att , used to attract the ship towards the target point;

[0071]

[0072] Among them, katt is the gain coefficient of the gravitational potential field; q is the current position of the ship; q goal is the target position;

[0073] A2: Define the obstacle repulsive potential field function U using Equation (16) rep , which generates a repulsive force when the ship approaches the obstacle;

[0074]

[0075] where k rep is the gain coefficient of the repulsive potential field; q obs is the position of the obstacle; d0 is the action range of the repulsive potential field;

[0076] A3: Perform negative gradient operations on the gravitational potential field function and the repulsive potential field function through Equations (17) and (18) respectively to obtain the gravitational and repulsive force functions;

[0077]

[0078] A4: Superimpose the repulsive forces of all obstacles using Equation (19) to obtain the resultant force F at the current position of the ship total ;

[0079]

[0080] where m is the number of obstacles acting on the ship;

[0081] A5: Add F total to the state space S, so that the deep reinforcement learning can learn the obstacle avoidance strategy more accurately in a complex environment.

[0082] As a preferred technical solution of the present invention, the prioritized experience replay in step S5-3 specifically includes the following steps:

[0083] The experience obtained after the agent interacts with the environment is stored in the experience pool;

[0084] Use Equation (20) to sort the experience samples according to the TD error size and adjust their sampling probabilities, and preferentially select the samples that have a more significant impact on the training benefit;

[0085]

[0086] where P(i) is the sampling probability; k is the total number of samples; α ∈ [0,1], which is an adjustable parameter that controls the randomness and bias of the sampling process;

[0087] Correct the bias caused by non-uniform sampling through Equation (21), introduce the importance sampling weight, and ensure that the training process remains unbiased;

[0088]

[0089] Among them, N is the total number of cached experiences; β is a parameter for adjusting the importance sampling weight intensity.

[0090] As a preferred technical solution of the present invention, the adaptive exploration strategy in step S5-4 specifically includes the following steps:

[0091] Design a non-linear decay ε-greedy strategy using Equation (22). In the initial stage, let the exploration probability ε take a larger value to cover different environmental states;

[0092] ε = max(ε min , ε max - k * ε decay ) (22)

[0093] Among them, ε min is the minimum value of ε; ε max is the maximum value of ε; k is the number of iterations; ε decay is the decay coefficient;

[0094] Gradually decay the value of ε with the training iteration to improve the utilization rate of historical experience and stably converge to the optimal strategy.

[0095] As a preferred technical solution of the present invention, the network parameter update in step S5-5 specifically includes the following steps:

[0096] In each training cycle, batch sample experiences (s, a, r, s′) from the experience pool, perform gradient backpropagation through the loss function, and update the current Q-network parameter θ; where s is the state, α is the learning rate, r is the immediate reward, and s′ is the next state;

[0097] Every preset fixed number of steps, synchronize θ to the target Q-network parameter θ-, to maintain the stability of the target Q-network;

[0098] Loop and iterate the above process until a converged deep reinforcement learning model is obtained.

[0099] As a preferred technical solution of the present invention, when performing deep reinforcement learning training in step S5, the following DQN structure is used to estimate the Q value corresponding to the state s and the action a, specifically including:

[0100] Based on the update mechanism of Q-Learning: Use the Q-value iteration method shown in Equation (23) to perform Bellman recursive update on Q(s, a) when executing action a in state s;

[0101] Q(s, a) ← Q(s, a) + α(r + γmax a′(Q(s′, a′) - Q(s, a)) (23)

[0102] where Q(s, a) is the expected cumulative reward obtained after executing action a in state s; α is the learning rate; r is the immediate reward; max a′ Q(s′, a′) is the Q - value of the optimal action selected in the next state s′;

[0103] Target network and main network: The neural network parameters are divided into main network parameters and target network parameters, and the target Q - value y is calculated using Equation (24);

[0104] y = r + γmax a′ Q(s′, a′; θ - ) (24)

[0105] where θ - is the target network parameter; max a′ Q(s′, a′; θ - ) is the maximum Q - value in the next state s′;

[0106] Mean - square error loss function: Equation (25) is used to measure the difference between the predicted Q - value and the target Q - value; Equation (26) is used to update the main network parameter θ through gradient descent;

[0107] L(θ) = E[(y - Q(s, a; θ)) 2 (25)

[0108]

[0109] where L(θ) is the loss function; Q(s, a; θ) is the Q - value predicted by the current Q - network; η is the learning rate; is the gradient of L(θ) with respect to θ; E[·] is the expectation operation;

[0110] Experience replay mechanism: The experiences obtained by the agent interacting with the environment are stored in the experience pool; Each time during training, a small batch of interaction data is randomly sampled from the experience pool for network update, thereby breaking the temporal correlation between samples and improving the training efficiency and stability.

[0111] Compared with the related prior art, the beneficial effects of the present invention are:

[0112] Fully meet the maritime collision avoidance rules: By embedding the constraints of the International Regulations for Preventing Collisions at Sea (COLREGS) into the deep reinforcement learning algorithm, the present invention enables the intelligent agent to make compliant decisions in the face of various collision avoidance scenarios, significantly reducing the collision risk and improving safety.

[0113] Improving Learning Efficiency and Stability: The present invention combines the improved structures of Double DQN and Dueling DQN, effectively alleviating the problem of overestimation of Q-values and accelerating the convergence speed of training. In addition, through the Prioritized Experience Replay (PER) mechanism, the attention to important experience samples is strengthened, further enhancing the efficiency and stability of training.

[0114] Comprehensive Reward Function Ensuring Safety and Efficiency: The present invention designs a comprehensive reward function including various rewards, such as distance reward, obstacle and boundary reward, end point reward, dynamic collision avoidance reward, and turning reward, etc. This helps to reach the target point quickly and smoothly while ensuring the safety of the path.

[0115] Artificial Potential Field Assisting in Precise Collision Avoidance: The present invention introduces an artificial potential field force in the state space, taking the resultant force of the target attraction and the obstacle repulsion as an important feature, improving the sensitivity and global nature to the obstacle distribution, enabling the ship to learn a reasonable route strategy faster.

[0116] Adaptive Greedy Strategy and B-spline Smoothing to Optimize the Trajectory: Dynamically adjust the balance between exploration and exploitation through the adaptive greedy (ε-greedy) strategy, and combine B-spline curve to smooth the discrete planning path, reducing the redundant energy consumption and navigation risk of sharp turns, while improving the overall efficiency and stability of navigation. Description of the Drawings

[0117] Figure 1 is a flowchart of a ship path planning method based on deep reinforcement learning according to the present invention;

[0118] Figure 2 is a meeting situation division diagram and corresponding obstacle avoidance strategy diagram of the embodiment provided by the present invention;

[0119] Figure 3 is a diagram of the force situation in the potential field of the embodiment provided by the present invention;

[0120] Figure 4 is a state space diagram of the embodiment provided by the present invention;

[0121] Figure 5 is an action space diagram of the embodiment provided by the present invention;

[0122] Figure 6 are the path planning results of four algorithms of the embodiment provided by the present invention, where Figure 6 (a) is the simulation result of Environment 1, Figure 6 (b) is the simulation result of Environment 2;

[0123] Figure 7 is a comparison diagram of the change of D3QN and ID3QN-PER reward values of the embodiment provided by the present invention, where, Figure 7(a) is the comparison chart of D3QN changes, Figure 7 (b) is the comparison chart of ID3QN-PER changes;

[0124] Figure 8 It is the collision avoidance result chart of the embodiment provided by the present invention. Among them, Figure 8 (a) is the overtaking scenario, Figure 8 (b) is the port side crossing encounter scenario, Figure 8 (c) is the starboard side crossing encounter scenario, Figure 8 (d) is the head-on encounter scenario;

[0125] Figure 9 It is the collision avoidance trajectory chart of multi-obstacle path planning of the embodiment provided by the present invention. Detailed implementation manners

[0126] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. However, the present invention can be implemented in many different ways and should not be construed as limited to the embodiments shown; on the contrary, these embodiments provide implementation manners that meet the applicable legal requirements for those skilled in the art.

[0127] Embodiment 1: As Figure 1 shown, this embodiment provides a ship path planning method based on deep reinforcement learning, which can not only follow the international collision avoidance rules in the maritime dynamic path planning, improve the learning efficiency and stability in the high-dimensional state environment, but also generate smooth and energy-saving navigation trajectories, and solve the problems such as low training efficiency and unstable paths that are prone to occur in the traditional DQN for ship path planning. The specific steps are as follows:

[0128] S1: Environment modeling and encounter situation recognition, which specifically includes the following steps:

[0129] S1-1: Discretize the modeling of the target sea area to obtain a feasible area; at each discrete moment t, let the current position of the ship be (x t , y t ), and define the next movable position of the ship according to Equation (1);

[0130] P t+1 = {(x, y)|(x, y) ∈ G f , x ∈ [x t - 1, x t + 1], y ∈ [y t - 1, y t + 1]} (1)

[0131] Among them, G f is the ship's feasible area; x t , y t are the abscissa and ordinate of the ship's current position respectively;

[0132] S1-2: For static obstacles such as reefs and shipwrecks and dynamic obstacles such as other moving ships navigating in the sea environment, monitor and update the distribution information of the obstacles in the feasible area in real time;

[0133] S1-3: Divide the encounter situations of ships in the sea area according to the COLREGS regulations, and design corresponding obstacle avoidance strategies based on the relative course and speed relationship; There are three encounter situations for ships: head-on situation, overtaking situation, and crossing situation, and the corresponding avoidance directions are given. According to the relative positions of the two ships, it is divided into four obstacle avoidance strategies. The encounter situation division diagram and the corresponding obstacle avoidance strategies are as Figure 2 described, specifically divided into:

[0134] In the head-on situation, when the angle between one's own ship and the target ship is in [355°, 360°] or [0°, 5°], both sides should change their own headings and sail to the starboard side, so that each ship passes from the port side of the other ship. In the overtaking situation, when one's own ship is sailing in a certain direction 22.5° behind the target ship, one's own ship is regarded as overtaking and should give way to the target ship. In the crossing situation, except for the above two situations of head-on situation and overtaking situation, other situations are regarded as crossing situations. Among them, the crossing situation is divided into two situations. The first is that the target ship is located on the starboard side of one's own ship, then this ship should take avoidance operations and sail to the right; the second is that one's own ship is located on the starboard side of the target ship, then the target ship should take avoidance operations and this ship sails normally.

[0135] S2: State space construction and artificial potential field enhancement, specifically including the following steps:

[0136] S2-1: Compose the key information that can be obtained by the ship in the sea environment into the state space S, including the current position (x t , y t ) of the ship, the distance between the ship and the target point, the current heading angle of the ship, and the relative positions of other dynamic ships in the environment;

[0137] S2-2: As Figure 3 shown, introduce an artificial potential field to strengthen the description of the environment, including the following steps:

[0138] A1: Use Equation (15) to define the target attraction potential field function U att to attract the ship towards the target point;

[0139]

[0140] Among them, k att is the gravitational potential field gain coefficient; q is the current position of the ship; q goal is the target position;

[0141] A2: Define the obstacle repulsive potential field function \(U\) using Equation (16) rep , which generates a repulsive force when the ship approaches an obstacle;

[0142]

[0143] where \(k\) rep is the repulsive potential field gain coefficient; \(q\) obs is the position of the obstacle; \(d_0\) is the action range of the repulsive potential field;

[0144] A3: Perform negative gradient operations on the gravitational potential field function and the repulsive potential field function through Equations (17) and (18) respectively to obtain the gravitational and repulsive force functions;

[0145]

[0146] A4: Superimpose the repulsive forces of all obstacles using Equation (19) to obtain the resultant force \(F\) at the current position of the ship total ;

[0147]

[0148] where \(m\) is the number of obstacles acting on the ship;

[0149] A5: Add \(F\) total to the state space \(S\) so that the deep reinforcement learning can learn the obstacle avoidance strategy more accurately in a complex environment.

[0150] S2-3: As Figure 4 shown, the final state space \(S\) satisfies Equation (2), ensuring a comprehensive description of the dynamic changes of the ship itself and its surrounding environment, where Equation (2) is:

[0151] S = {x ship , y ship , d se , step ship , θ ship , f x , f y , x obs , y obs} (2)

[0152] where \(x\) ship and \(x\) ship represent the current position of the ship; \(d\) se represents the distance between the ship and the end point; step ship represents the current number of steps of the ship; θ ship represents the current angle of the ship; f x and f y respectively represent the potential field forces acting on the ship on the x-axis and y-axis; \(x\) obs and yobs Indicates the position of the current dynamic obstacle;

[0153] S3: Action space design, as Figure 5 shown, the heading adjustment operation of the ship is discretized to obtain the action set A, which satisfies Equation (3); each discrete action represents the change in the steering angle relative to the current heading, including keeping the heading unchanged, small-angle steering, medium-angle steering, and large-angle steering, where Equation (3) is:

[0154] A = {0°, 15°, -15°, 30°, -30°, t5°, -t5°, 60°, -60°} (3)

[0155] Among them, 0° indicates that the current heading remains unchanged; 15° and -15° indicate small adjustments to the ship's direction, suitable for fine-tuning the heading; 30° and -30° indicate medium-angle steering adjustments, suitable for avoiding relatively close obstacles and can achieve obvious direction correction; 60° and -60° indicate large-angle heading adjustments, used to cope with emergency obstacle avoidance or large-angle steering requirements;

[0156] S4: Comprehensive reward function design, define the total reward function R as shown in Equation (4), and the specific calculation formula is:

[0157] R t = R τ1 + R τ2 + R τ3 + R τt + R τ5 (4)

[0158] Among them, R τ1 is the distance reward function; R τ2 is the steering reward function; R τ3 is the obstacle and boundary reward function; R τt is the end point reward function; R τ5 is the dynamic collision avoidance reward function. To enable the ship to balance safety and efficiency during path planning, it specifically includes the following characteristics:

[0159] The distance reward function judges whether the ship is approaching the target area according to the change in the distance between the ship and the target point; if the ship is closer to the target, the positive reward is greater; if the distance increases, the reward decreases; the calculation formula is:

[0160] R τ1 = k1·(d t - d t+1 ) (5)

[0161] Among them, k1 is the weight coefficient; d t is the distance between the ship and the target point at time t; d t+1is the distance between the ship and the target point at time t+1;

[0162] The turning reward applies a moderate negative penalty to frequent or large-angle unnecessary turns, guiding the ship to maintain a stable course as much as possible under feasible conditions; the calculation formula is:

[0163] R τ2 = k2·cos(3*θ t ) (6)

[0164] where k2 is the weight coefficient; θ t indicates the turning angle of the ship's movement at the next moment;

[0165] The obstacle and boundary reward gives a negative reward when the ship approaches an obstacle or exceeds the environmental boundary, to constrain the safe navigation of the ship; the calculation formula is:

[0166]

[0167] where k3 is a negative constant; d o is the minimum distance between the ship and the obstacle or boundary;

[0168] The end point reward gives an additional high positive reward when the ship successfully reaches the target position, to highlight the priority of task completion; the calculation formula is:

[0169] R τt = k t (8)

[0170] where k t is a positive constant;

[0171] The dynamic collision avoidance reward determines the encounter situation based on information such as the relative position, speed, and course between the own ship and the target ship; the calculation formula is:

[0172]

[0173] where r ho is the reward for the head-on encounter situation; r ot is the reward for the overtaking situation; r cleft is the reward for the port side crossing; r cright is the reward for the starboard side crossing; a ta is the action to be taken by the own ship in the next step;

[0174] S5: Training based on the depth reinforcement learning of ID3QN-PER, specifically including the following structural features:

[0175] S5-1: Use a double-network structure that uses two neural networks with the same structure but independent parameters as the current Q-network and the target Q-network respectively, and calculate the target Q-value through Equation (10):

[0176]

[0177] Among them, represents the target network; argmax a′ Q(s′, a′, θ) is the optimal action generated by the current network;

[0178] S5-2: Use a dueling network structure that splits the Q-value into a state value function V(s) and an advantage function A(s, a), and combines them through Equation (11):

[0179]

[0180] Among them, V(s) is the value of state s; A(s, a) is the relative advantage of action a in state s; |A| is the number of actions; is the average advantage value of all actions;

[0181] S5-3: Prioritized experience replay, prioritize and sample historical experiences based on TD errors to improve the utilization efficiency of key samples; the specific steps of prioritized experience replay in Step S5-3 are as follows:

[0182] The experiences obtained after the agent interacts with the environment are stored in the experience pool;

[0183] Use Equation (20) to sort the experience samples according to the TD error size and adjust their sampling probabilities, and preferentially select samples that have a more significant impact on the training benefit;

[0184]

[0185] Among them, P(i) is the sampling probability; k is the total number of samples; α ∈ [0, 1], which is an adjustable parameter that controls the randomness and bias of the sampling process;

[0186] Correct the bias caused by non-uniform sampling through Equation (21), introduce importance sampling weights, and ensure that the training process remains unbiased;

[0187]

[0188] Among them, N is the total number of cached experiences; β is a parameter that adjusts the intensity of the importance sampling weights, and the weights will gradually approach 1 as the training progresses, reducing the impact and improving the stability of the training.

[0189] S5-4: Adaptive exploration strategy, which balances exploration and exploitation in a non-linear decaying ε-greedy manner, expands the search scope in the early stage of training, and gradually utilizes the existing experience in the later stage; the adaptive exploration strategy in step S5-4 specifically includes the following steps:

[0190] Design a non-linear decaying ε-greedy strategy using Equation (22). In the initial stage, set the exploration probability ε to a relatively large value to cover different environmental states;

[0191] ε = max(ε min , ε max - k * ε decay )(22)

[0192] where ε min is the minimum value of ε; ε max is the maximum value of ε; k is the number of iterations; ε decay is the decay coefficient;

[0193] Gradually decay the value of ε with the training iterations, improve the utilization rate of historical experience, and stably converge to the optimal strategy.

[0194] S5-5: Network parameter update, periodically sample a small batch of data from the experience pool for gradient descent, and synchronize the current Q-network parameters to the target Q-network until a convergent policy model is obtained; the network parameter update in step S5-5 specifically includes the following steps:

[0195] In each training cycle, batch sample the experience (s, a, r, s') from the experience pool, perform gradient backpropagation through the loss function, and update the current Q-network parameters θ; where s is the state, α is the learning rate, r is the immediate reward, and s' is the next state;

[0196] Every preset fixed number of steps, synchronize θ to the target Q-network parameters θ-, to maintain the stability of the target Q-network;

[0197] Loop and iterate the above process until a convergent deep reinforcement learning model is obtained.

[0198] S6: Path generation and B-spline smoothing, specifically including the following steps:

[0199] S6-1: After the ID3QN-PER algorithm is trained, the optimal action a can be quickly inferred from the trained Q-network according to the current state S during actual navigation to achieve real-time obstacle avoidance and navigation;

[0200] S6-2: For the discrete navigation path output by the algorithm, extract the turning points as the control points of the B-spline curve. The calculation formula of the B-spline curve is:

[0201]

[0202] Among them, C(u) is the point on the curve; n is the number of control points; B i is the control point; u is the knot vector, which controls the shape of the basis function and the segmentation of the curve; N i,k (u) represents the k-th order B-spline basis function;

[0203] S6-3: Interpolate the control points into a continuous curve, and recursively define the piecewise polynomial of the B-spline basis function in combination with Equations (13) and (14) to achieve smooth generation of the path:

[0204]

[0205] Among them, x i is the element in the knot vector;

[0206] S6-4: Evaluate the route smoothed by the B-spline. If there is a conflict with dynamic obstacles or environmental changes, return to step S5 to re-train and update the strategy;

[0207] S7: Navigation execution and adaptive update, specifically including the following steps:

[0208] S7-1: Take the smoothed path as the final route for the ship to execute, and perform actual navigation through the ship's own steering and speed control system;

[0209] S7-2: Monitor the sea conditions in real time during navigation. Once it is detected that the situation of new ships or obstacles has changed significantly, repeat the above steps S2 to S6, and adjust the strategy and path again in combination with the ID3QN-PER algorithm to ensure compliance with the COLREGS regulations at all times in a dynamic environment and improve navigation efficiency and safety.

[0210] Embodiment 2: To verify the feasibility of the ID3QN-PER algorithm. In a static simulation environment, this embodiment designs two 50×50 visual simulation environments, which only contain static obstacles at sea; in a dynamic simulation environment, this embodiment first designs a sea collision avoidance rule experiment to simulate the collision avoidance behavior of a ship when encountering other moving ships, and at the same time designs a comprehensive experiment including dynamic ships and sea obstacles to examine the adaptability and effect of the algorithm in the face of various obstacles.

[0211] In the static simulation environment, only the static obstacles in the map are included to evaluate the navigation and path planning capabilities of the algorithm. These obstacles are set as regions with different shapes and sizes. The algorithm needs to identify and avoid these obstacles and plan a safe path to the end point. The comparison algorithms include the A* algorithm, the artificial potential field (APF) algorithm, and the duel double deep Q-learning (D3QN) algorithm. In this embodiment, the performance of the path planning results of different algorithms is calculated from four aspects: path length, number of corners, accumulated heading angle, and minimum obstacle collision distance to evaluate the superiority and robustness of the proposed ID3QN-PER algorithm. Now, 10 experiments are respectively carried out on D3QN and the algorithm of the present invention, and the average value of the path length is taken. The path planning results of the four algorithms are as Figure 6 shown, and the comparison of evaluation indexes is shown in Table 1, and the comparison of the change of reward values is as Figure 7 shown.

[0212] Table 1: Comparison of evaluation indexes

[0213]

[0214] Figure 7 shows the change of the reward value during the training process of the D3QN and ID3QN-PER algorithms. Since the ID3QN-PER algorithm introduces APF and PER compared with the D3QN algorithm, it has higher efficiency and better performance in dealing with path planning and obstacle avoidance tasks in complex environments.

[0215] The ship collision avoidance experiment aims to verify whether ID3QN-PER can perform path planning according to the maritime collision avoidance rules during maritime movement. The four COLREGS situations and collision avoidance results are as Figure 8 shown. Among them, the blue straight line is the route of one's own ship, the red straight line is the route of the target ship, and each arrow indicates that the target ship moves once. Figure 8 (a) is the overtaking scenario. This scenario is caused by one's own ship actively, so it is set that the speed of one's own ship is twice that of the target ship. In the remaining scenarios, the speeds of one's own ship and the target ship are set to be the same. When one's own ship and the target ship are in the overtaking scenario, the direction of the target ship is set to face one's own ship, and one's own ship overtakes from the left side of the target ship. Figure 8 (b) is the starboard crossing scenario. When one's own ship discovers that there is a target ship passing by on the starboard side, one's own ship should sail normally. However, for safety reasons, one's own ship can turn right to leave a safe distance and then sail. Figure 8 (c) is the port crossing scenario. When one's own ship discovers that there is a target ship passing by on the port side, one's own ship should take the measure of turning right and pass behind the target ship after ensuring safety. Figure 8 (d) is the head-on scenario. The direction of the target ship is set to be the same as that of one's own ship. One's own ship turns to the right according to the COLREGS rules and avoids passing from the left side of the target ship.

[0216] According to the COLREGS collision avoidance rules, a multi-obstacle path planning collision avoidance experiment is designed in this embodiment to test the algorithm of the present invention. Now, two moving ships are added to the environment. Among them, the speed of the own ship is twice that of the moving ships. The yellow dotted line represents the trajectory without the influence of the moving ships, and the blue solid line represents the moving trajectory after adding two moving ships. The multi-obstacle path planning collision avoidance trajectory is as Figure 9 shown.

[0217] Figure 9 It shows the collision avoidance path planning trajectory when the own ship encounters two moving ships at the same time when sailing towards the target. By comparing the original trajectory with the trajectory after adding the moving ships, it can be seen that when the own ship approaches the red target ship, it is judged that the own ship and the red target ship are in an overtaking scenario, and overtaking is actively carried out from the right side. When the own ship approaches the green ship, it is judged that the own ship and the green target ship are in a starboard crossing encounter scenario, and a right turn is taken to actively avoid the target ship. The collision avoidance behavior complies with the COLREGS rules.

[0218] From the above results, it can be seen that the practicability and robustness of the ID3QN-PER algorithm are verified in both static and dynamic simulation environments in this embodiment. Compared with the traditional D3QN and other control algorithms, ID3QN-PER not only has more advantages in terms of training convergence speed, path smoothness and obstacle safety distance, but also can strictly follow the COLREGS to achieve effective avoidance among multiple ships. In summary, the feasibility and superior performance of the present invention in the offshore environment are illustrated, providing strong support for the efficient planning and safe navigation of ships in complex navigation environments.

[0219] The above embodiments only represent several implementation manners of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. A ship path planning method based on deep reinforcement learning, characterized in that: The method includes the following steps: S1: Environmental modeling and encounter situation recognition, specifically including the following steps: S1-1: Discretize and model the target sea area to obtain a feasible region; at each discrete time t, let the current position of the ship be (x t , y t ), and define the next movable position of the ship according to Equation (1); P t+1 = {(x, y)|(x, y) ∈ G f , x ∈ [x t -1, x t +1], y ∈ [y t -1, y t +1]} (1) Among them, G f is the feasible region of the ship; x t , y t are the abscissa and ordinate of the current position of the ship respectively; S1-2: For static obstacles such as reefs and shipwrecks and other moving ships (dynamic obstacles) navigating in the sea surface environment, monitor and update the distribution information of the obstacles in the feasible area in real time; S1-3: Divide the encounter situations of ships in the sea area according to the COLREGS regulations, and design corresponding obstacle avoidance strategies according to the relative course and speed relationship; S2: State space construction and artificial potential field enhancement, specifically including the following steps: S2-1: Compose the state space S from the key information that can be obtained by the ship in the sea environment, including the current position (x t , y t ) of the ship, the distance between the ship and the target point, the current heading angle of the ship, and the relative positions of other dynamic ships in the environment; S2-2: Introduce an artificial potential field to strengthen the description of the environment; S2-3: The final state space S satisfies Equation (2), ensuring a comprehensive description of the dynamic changes of the ship itself and the surrounding environment, where Equation (2) is: S = {x ship , y ship , d se , step ship , θ ship , f x , f y , x obs , y obs} (2) where x ship and y ship represent the current position of the ship; d se represents the distance between the ship and the end point; step ship represents the current number of steps of the ship; θ ship represents the current angle of the ship; f x and f y respectively represent the potential field forces acting on the ship in the x-axis and y-axis directions; x obs and y obs represent the position of the current dynamic obstacle. S3: Action space design, discretize the course adjustment operation of the ship to obtain an action set A that satisfies Equation (3); each discrete action represents the change in the steering angle relative to the current course, including keeping the course unchanged, small-angle steering, medium-angle steering, and large-angle steering, where Equation (3) is: A＝{0°,15°,-15°,30°,-30°,45°,-45°,60°,-60°} (3) Among them, 0° means keeping the current course unchanged; 15° and -15° mean small adjustments to the ship's direction, suitable for fine-tuning the course; 30° and -30° mean medium-angle steering adjustments, suitable for avoiding relatively close obstacles and can achieve obvious direction correction; 60° and -60° mean large-course adjustments, used to cope with emergency obstacle avoidance or large-angle steering requirements; S4: Comprehensive reward function design, define the total reward function R as shown in Equation (4), and the specific calculation formula is: R t = R τ1 + R τ2 + R τ3 + R τ4 + R τ5 (4) where, R τ1 is the distance reward function; R τ2 is the steering reward function; R τ3 is the obstacle and boundary reward function; R τ4 is the end point reward function; R τ5 is the dynamic collision avoidance reward function. To enable the ship to balance safety and efficiency during path planning, it specifically includes the following features: The distance reward function judges whether the ship is approaching the target area according to the change in the distance between the ship and the target point; if the ship is closer to the target, the positive reward is greater; if the distance increases, the reward decreases; the calculation formula is: R τ1 = k1·(d t - d t+1 ) (5) where k1 is a weight coefficient; d t is the distance between the ship and the target point at time t; d t+1 is the distance between the ship and the target point at time t+1; The steering reward imposes a moderate negative penalty on frequent steering or large-angle unnecessary steering, guiding the ship to keep the course as stable as possible under feasible conditions; the calculation formula is: R τ2 = k2·cos(3*θ t ) (6) where k2 is a weight coefficient; θ t indicates the steering angle of the ship's movement at the next moment; The obstacle and boundary reward gives a negative reward when the ship approaches an obstacle or exceeds the environmental boundary, so as to restrict the safe navigation of the ship; the calculation formula is: where k3 is a negative constant; d o is the minimum distance between the ship and the obstacle or boundary, and d s is the actual distance between the ship and the obstacle or boundary; The end reward gives an additional high positive reward when the ship successfully reaches the target position, highlighting the priority of task completion; the calculation formula is: R τ4 = k4(8) Among them, k4 is a positive constant; The dynamic collision avoidance reward determines the encounter situation according to the relative position, speed, and course information between the own ship and the target ship; the calculation formula is: Among them, r ho is the reward for head-on encounter situation; r ot is the reward for overtaking situation; r cleft is the reward for port side crossing; r cright is the reward for starboard side crossing; a ta is the action to be taken by one's own ship in the next step; θ own is the angle between the traveling direction of one's own ship and the x-axis direction; θ tar is the angle between the traveling direction of the target ship and the x-axis direction; S5: Deep reinforcement learning training based on ID3QN-PER, specifically including the following structural features: S5-1: Use a dual-network structure, which uses two neural networks with the same structure but independent parameters as the current Q network and the target Q network respectively, and calculates the target Q value through Equation (10): Among them, represents the target network; argmax a' Q(s', a', θ) is the optimal action generated by the current network; r is the immediate reward; γ is the discount factor; S5-2: Use a dueling network structure, which splits the Q value into a state value function V(s) and an advantage function A(s,a), and combines them through Equation (11): Among them, V(s) is the value of state s; A(s,a) is the relative advantage of action a in state s; |A| is the number of actions; is the average advantage value of all actions; S5-3: Prioritized experience replay, prioritize and sample historical experiences based on the TD error, and improve the utilization efficiency of key samples; S5-4: Adaptive exploration strategy, which balances exploration and exploitation through a nonlinearly decaying ε-greedy approach, expanding the search range in the early stages of training and gradually leveraging existing experience in the later stages; S5-5: Network parameter update, periodically sampling small batches of data from the experience pool for gradient descent, and synchronizing the current Q network parameters to the target Q network until a converged policy model is obtained; S6: Path generation and B-spline smoothing, specifically including the following steps: S6-1: After the ID3QN-PER algorithm is trained, the trained Q network can quickly infer the optimal action a based on the current state S during actual navigation, achieving real-time obstacle avoidance and navigation. S6-2: For the discrete navigation path output by the algorithm, the turning points are extracted as the control points of the B-spline curve. The calculation formula of the B-spline curve is: Among them, C(u) is the point on the curve; n is the number of control points; B i is the control point; u is the knot vector, which controls the shape of the basis function and the segmentation of the curve; N i,k (u) represents the k-th order B-spline basis function; S6-3: The control points are interpolated into a continuous curve, and the B-spline basis function is recursively defined as a piecewise polynomial by combining equations (13) and (14) to achieve smooth path generation: where x i is an element in the node vector; S6-4: Evaluate the B-spline smoothed route. If there is a conflict with a dynamic obstacle or environmental change, return to step S5 to retrain and update the strategy. S7: Navigation execution and adaptive update, specifically including the following steps: S7-1: The smoothed path is used as the final route for the ship to navigate using the ship's own steering and speed control systems. S7-2: Monitor sea conditions in real time during navigation. Once a new ship is detected or the obstacle situation changes significantly, repeat steps S2 to S6 above and adjust the strategy and path again in combination with the ID3QN-PER algorithm to ensure that COLREGS specifications are always met in a dynamic environment and to improve navigation efficiency and safety.

2. The method for ship path planning based on deep reinforcement learning according to claim 1, characterized in that: The COLREGS specification described in step S1-3 divides encounter situations into head-on encounter, overtaking, and crossing encounter situations, and further subdivides them into port crossing and starboard crossing. Corresponding obstacle avoidance strategies are designed based on the relative heading and speed relationship. Among them, the friendly ship should actively make an evasive move when it is determined to be an overtaking scenario. In the head-on encounter scenario, both parties will simultaneously travel to the starboard side. In the crossing scenario, the ship on the starboard side will have priority. This standardizes the course adjustment principles in different situations.

3. The method for ship path planning based on deep reinforcement learning according to claim 1, characterized in that: In step S2-2, an artificial potential field is introduced to enhance the characterization of the environment, which specifically includes the following steps: A1: Define the target attraction potential field function U using Equation (15) att to attract the ship towards the target point; where k att is the gravitational potential field gain coefficient; q is the current position of the ship; q goal is the target position; A2: Define the obstacle repulsive potential field function U using Equation (16) rep , which generates a repulsive force when the ship approaches an obstacle; where k rep is the repulsive potential field gain coefficient; q obs is the obstacle position; d0 is the action range of the repulsive potential field; A3: The gravitational and repulsive force functions can be obtained by performing negative gradient operations on the gravitational potential field function and the repulsive potential field function through equations (17) and (18) respectively; A4: Superimpose the repulsive forces of all obstacles using Equation (19) to obtain the resultant force F at the current position of the ship total ; Where m is the number of obstacles acting on the ship; A5: Add F total to the state space S, enabling the deep reinforcement learning to more accurately learn the obstacle avoidance strategy in complex environments.

4. The method for ship path planning based on deep reinforcement learning according to claim 1, wherein: The priority experience playback in step S5-3 specifically includes the following steps: The experience gained by the agent after interacting with the environment is stored in the experience pool; Use formula (20) to sort the empirical samples according to the TD error and adjust their sampling probability, giving priority to samples that have a more significant impact on training benefits; Where P(i) is the sampling probability; k is the total number of samples; α∈[0,1] is an adjustable parameter that controls the randomness and bias of the sampling process; The bias caused by non-uniform sampling is corrected through Equation (21), and importance sampling weights are introduced to ensure the unbiasedness of the training process; where N is the total number of cached experiences; β is a parameter for adjusting the intensity of importance sampling weights.

5. A ship path planning method based on deep reinforcement learning according to claim 1, characterized in that: The adaptive exploration strategy in step S5-4 specifically includes the following steps: Design a non-linear decaying ε-greedy strategy using Equation (22). Let the exploration probability ε take a larger value in the initial stage to cover different environmental states; ε = max(ε min , ε max -k * ε decay ) (22) Among them, ε min is the minimum value of ε; ε max is the maximum value of ε; k is the number of iterations; ε decay is the attenuation coefficient; Gradually decay the value of ε as the training iteration progresses to improve the utilization rate of historical experiences and stably converge to the optimal strategy.

6. The method for ship path planning based on deep reinforcement learning according to claim 1, wherein: The update of network parameters in step S5-5 specifically includes the following steps: In each training cycle, sample experiences (s, a, r, s') in batches from the experience pool, perform gradient backpropagation through the loss function, and update the current Q-network parameters θ; where s is the state, α is the learning rate, r is the immediate reward, and s' is the next state; Every preset fixed number of steps, synchronize θ to the target Q-network parameters θ-, to maintain the stability of the target Q-network; Iterate the above process in a loop until a converged deep reinforcement learning model is obtained.

7. A ship path planning method based on deep reinforcement learning according to claim 1, characterized in that: During the deep reinforcement learning training in step S5, the following DQN structure is used to estimate the Q value corresponding to the state s and the action a, specifically including: Q-Learning based update mechanism: Using the Q-value iteration method shown in Equation (23), when executing action a in state s, perform Bellman recursive update on Q(s,a); Q(s,a)←Q(s,a)+α(r+γmaxa ′ Q(s′,a′)-Q(s,a)) (23) Among them, Q(s,a) is the expected cumulative reward obtained after executing action a in state s; α is the learning rate; r is the immediate reward; max a' Q(s',a') is the Q value of the optimal action selected in the next state s'; Target network and main network: Divide the neural network parameters into main network parameters and target network parameters, and calculate the target Q value y using Equation (24); y = r + γmax a' Q(s′, a′; θ - ) (24) where θ - is the target network parameter; max a' Q(s', a'; θ - ) is the maximum Q value at the next state s'. Mean squared error loss function: Use Equation (25) to measure the difference between the predicted Q value and the target Q value; use Equation (26) to update the main network parameters θ through gradient descent; L(θ) = E[(y - Q(s, a; θ)) 2 (25) Among them, \(L(\theta)\) is the loss function; \(Q(s,a;\theta)\) is the Q-value predicted by the current Q-network; \(\eta\) is the learning rate; is the gradient of \(L(\theta)\) with respect to \(\theta\); \(E[\cdot]\) is the expectation operation; Experience replay mechanism: The experiences obtained by the agent interacting with the environment are stored in the experience pool; each time during training, randomly sample a small batch of interaction data from the experience pool for network update, thereby breaking the temporal correlation between samples and improving the training efficiency and stability.

Images