Unmanned ship dynamic path planning method and system based on deep reinforcement learning

Abstract

The present invention provides a method and system for dynamic path planning of an unmanned ship based on deep reinforcement learning, which relates to the field of path planning technology, including obtaining the initial position, target position and marine environment information of the unmanned ship; inputting the initial position, target position and marine environment information of the unmanned ship into a pre-constructed deep reinforcement learning model, and based on the deep reinforcement learning model, using a state value function to evaluate the long-term rewards of the unmanned ship taking different actions in various states, determining the optimal action strategy of the unmanned ship according to the long-term rewards, and obtaining a local path planning result of the unmanned ship; based on a preset path optimization algorithm, smoothly optimizing the local path planning result to obtain a dynamic global path of the unmanned ship; and according to the dynamic global path, controlling the heading and speed of the unmanned ship so that the unmanned ship travels along the dynamic global path to the target position.

Description

Technical Field

[0001] The present invention relates to path planning technology, and in particular to a method and system for dynamic path planning of an unmanned ship based on deep reinforcement learning. Background Art

[0002] With the continuous development of artificial intelligence technology, unmanned ships have been widely used in the fields of marine surveys, environmental monitoring, search and rescue, etc. Unmanned ships can autonomously navigate in complex marine environments and complete various tasks, with the advantages of high efficiency, low cost, and low risk. However, the dynamic and uncertain marine environment poses challenges to the path planning of unmanned ships. Traditional path planning methods are difficult to adapt to the dynamic changes of the marine environment, which limits the autonomy and intelligence of unmanned ships.

[0003] At present, common methods for unmanned ship path planning include graph search method, sampling method, optimization method, etc. A* algorithm is a classic heuristic graph search algorithm that selects the optimal path by evaluating the cost from the node to the target point.

[0004] However, the above traditional methods have certain limitations in dynamic marine environments. Graph search and sampling methods have difficulty in processing continuous state spaces and require discretization, resulting in high computational complexity; optimization methods are prone to fall into local optimality and are difficult to adapt to dynamic changes in the environment. In addition, these methods usually rely on pre-established environmental models and have difficulty processing unknown or uncertain environmental information. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for dynamic path planning of an unmanned ship based on deep reinforcement learning, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides a dynamic path planning method for unmanned ships based on deep reinforcement learning, including:

[0008] Obtaining the initial position, target position and ocean environment information of the unmanned ship, wherein the ocean environment information includes ocean current, wind speed, water depth and obstacle position; inputting the initial position, target position and ocean environment information of the unmanned ship into a pre-built deep reinforcement learning model, wherein the deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship by learning and training the sample data of the dynamic path planning of the unmanned ship;

[0009] Based on the deep reinforcement learning model, a state value function is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states, and the optimal action strategy of the unmanned ship is determined according to the long-term rewards to obtain the local path planning result of the unmanned ship;

[0010] Based on a preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, the local path planning results are smoothly optimized to obtain the dynamic global path of the unmanned ship; according to the dynamic global path, the heading and speed of the unmanned ship are controlled so that the unmanned ship travels along the dynamic global path to the target position.

[0011] In an optional embodiment,

[0012] The initial position, target position and ocean environment information of the unmanned ship are input into a pre-built deep reinforcement learning model. The deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship through learning and training of the unmanned ship dynamic path planning sample data, including:

[0013] Encoding the initial position of the unmanned ship and the ocean environment information into a state vector as the input of a deep reinforcement learning model, wherein the deep reinforcement learning model includes an online network and a target network, wherein the online network is used to select an action and the target network is used to evaluate the Q value of the action;

[0014] According to the ε-greedy strategy, an online network is used to select actions, randomly selecting actions with ε probability, and selecting the action with the largest Q value with 1-ε probability. The ε value gradually decays during training;

[0015] The unmanned ship interacts with the environment according to the action with the largest Q value, and obtains an immediate reward and the next state. The immediate reward comprehensively considers the sailing distance, sailing time, energy consumption and obstacle avoidance factors;

[0016] The transfer samples consisting of the state, action, reward and next state are stored in a priority experience replay buffer, wherein the priority experience replay buffer determines its sampling priority according to the temporal difference error of the sample;

[0017] Sampling a batch of transfer samples from the priority experience replay buffer, calculating the temporal difference error of each transfer sample, using the target network to calculate the maximum Q value of the next state, and weighting the temporal difference error using the importance sampling weight to obtain a weighted mean square error loss function;

[0018] Based on the weighted mean square error loss function, the parameters of the online network are updated using a stochastic gradient descent algorithm, and the parameters of the online network are periodically copied to the target network; the above steps are repeated until the online network converges.

[0019] In an optional embodiment,

[0020] Calculating the maximum Q value of the next state using the target network includes:

[0021]

[0022] Among them, y represents the target Q value, r represents the immediate reward, γ represents the discount factor, a and θ represent the current selected action and the parameters of the Q network respectively, and s′ and θ - They represent the parameters of the next state and target network respectively.

[0023] In an optional embodiment,

[0024] Based on the deep reinforcement learning model, the state value function is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states, and the optimal action strategy of the unmanned ship is determined according to the long-term rewards. The local path planning results of the unmanned ship include:

[0025] The current position coordinates, heading angle and ocean environment information within the local sensing range of the unmanned ship are encoded as a state vector, and the next heading change and speed change of the unmanned ship are encoded as an action vector;

[0026] Using a dueling deep Q network to evaluate the long-term rewards of the unmanned ship taking different actions in different states, the dueling deep Q network includes two independent neural networks for estimating the state value function and the advantage function respectively;

[0027] The state value function represents the expected cumulative reward of the unmanned ship in the current state, and the advantage function represents the advantage of selecting a certain action over other actions in the current state;

[0028] The output of the state-value function network is combined with the output of the advantage function network through the aggregation layer to obtain the final state-action value function, which is used to evaluate the long-term reward of taking an action in the current state;

[0029] During the training process, the unmanned ship updates the parameters of the dueling deep Q network using the temporal difference learning algorithm through interaction with the environment;

[0030] At the decision-making moment, the unmanned ship uses the trained dueling deep Q network to calculate the long-term reward of each action based on the current state, selects the action with the largest reward as the optimal decision, and obtains the next heading and speed;

[0031] Through a continuous decision-making process, the unmanned ship obtains a local path from its current position to the target position.

[0032] In an optional embodiment,

[0033] The output of the state-value function network is combined with the output of the advantage function network through the aggregation layer to obtain the final state-action value function including:

[0034]

[0035] Among them, V(s) represents the state value function, A(s,a) represents the advantage function, |A| represents the size of the action space, s and a represent the current state and the currently selected action respectively, and a′ represents the next action.

[0036] In an optional embodiment,

[0037] Based on the preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, the local path planning results are smoothly optimized to obtain the dynamic global path of the unmanned ship, including:

[0038] Using the local path planning result of the unmanned ship obtained by deep reinforcement learning as the input of the path optimization algorithm, the local path planning result is determined by the current position and target position of the unmanned ship;

[0039] Constructing a time-varying artificial potential field of the unmanned ship in the marine environment, wherein the time-varying artificial potential field is composed of a gravitational potential field of the target point, a repulsive potential field of the obstacle, and a velocity potential field considering the speed loss of the unmanned ship;

[0040] Introducing a time-varying gravitational coefficient and a time-varying repulsive coefficient into the potential field functions of the gravitational potential field and the repulsive potential field, respectively, wherein the time-varying gravitational coefficient and the time-varying repulsive coefficient are dynamically adjusted according to the magnitude of the ocean current velocity;

[0041] Establishing the motion equation of the unmanned ship in the time-varying artificial potential field, taking the waypoints in the local path planning result as the initial trajectory, using the gradient descent algorithm to iteratively optimize the total potential energy function, and obtaining a dynamic global path;

[0042] The dynamic global path is output to the motion control module of the unmanned ship to guide the autonomous decision-making and motion control of the unmanned ship.

[0043] In an optional embodiment,

[0044] The gravitational potential field function of the target point is:

[0045]

[0046] Among them, U att (x) represents the gravitational potential field corresponding to the current position x of the unmanned ship, k att represents the gravitational coefficient, d goal Represents the Euclidean distance from the current position x of the unmanned ship to the target point;

[0047] The repulsive potential field function of the obstacle is:

[0048]

[0049] Among them, Urep (x) represents the repulsive potential field corresponding to the current position x of the unmanned ship, k rep represents the repulsion coefficient, d safe Indicates the safety distance threshold, d obs Indicates the shortest distance from the current position of the unmanned ship to the obstacle;

[0050] The time-varying attraction coefficient and time-varying repulsion coefficient are as follows:

[0051] k att (t) = att0*exp(-λ att V current (t));

[0052] k rep (t) = rep0*exp(-λ rep V current (t));

[0053] Among them, k att (t), k rep (t) represents the attraction coefficient and repulsion coefficient at time t, att0 and rep0 represent the initial attraction coefficient and initial repulsion coefficient, respectively. current (t) represents the ocean current velocity at time t;

[0054] The speed loss is given by the following formula:

[0055]

[0056] Among them, U vel (x) represents the speed loss value of the unmanned ship at position x, k vel represents the velocity loss coefficient, v max represents the maximum speed of the unmanned ship, and v(x) represents the current speed of the unmanned ship at position x;

[0057] The total potential energy function is given by the following formula:

[0058] U(x,t)=U att (x,t)+∑U rep (x,t)+U vel (x);

[0059] Among them, U(x,t) represents the total potential energy, U att (x, t) represents the gravitational potential field function corresponding to the current position x of the unmanned ship at time t, U rep (x, t) represents the repulsive potential field function corresponding to the current position x of the unmanned ship at time t, U vel (x) represents the speed loss value of the unmanned ship at position x.

[0060] In the second aspect of the embodiments of the present application,

[0061] Provides an unmanned ship dynamic path planning system based on deep reinforcement learning, including:

[0062] The first unit is used to obtain the initial position, target position and marine environment information of the unmanned ship, wherein the marine environment information includes ocean current, wind speed, water depth and obstacle position; the initial position, target position and marine environment information of the unmanned ship are input into a pre-built deep reinforcement learning model, and the deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship through learning and training of sample data of dynamic path planning of the unmanned ship;

[0063] The second unit is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states using a state value function based on the deep reinforcement learning model, determine the optimal action strategy of the unmanned ship according to the long-term rewards, and obtain a local path planning result of the unmanned ship;

[0064] The third unit is used to smoothly optimize the local path planning results based on a preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, to obtain a dynamic global path of the unmanned ship; according to the dynamic global path, the heading and speed of the unmanned ship are controlled so that the unmanned ship travels to the target position along the dynamic global path.

[0065] According to a third aspect of the embodiments of the present invention,

[0066] An electronic device is provided, comprising:

[0067] processor;

[0068] a memory for storing processor-executable instructions;

[0069] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0070] A fourth aspect of the embodiments of the present invention is:

[0071] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of a flow chart of a method for dynamic path planning of an unmanned ship based on deep reinforcement learning according to an embodiment of the present invention;

[0073] Figure 2 This is a schematic diagram of the structure of an unmanned ship dynamic path planning system based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0076] Figure 1 FIG. 1 is a flow chart of a method for dynamic path planning of an unmanned ship based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0077] S101. Obtaining the initial position, target position and marine environment information of the unmanned ship, wherein the marine environment information includes ocean current, wind speed, water depth and obstacle position; inputting the initial position, target position and marine environment information of the unmanned ship into a pre-built deep reinforcement learning model, wherein the deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship by learning and training the sample data of the dynamic path planning of the unmanned ship;

[0078] In an optional embodiment,

[0079] The initial position, target position and ocean environment information of the unmanned ship are input into a pre-built deep reinforcement learning model. The deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship through learning and training of the unmanned ship dynamic path planning sample data, including:

[0080] Encoding the initial position coordinates of the unmanned ship and the ocean environment information into a state vector as the input of a deep reinforcement learning model, wherein the deep reinforcement learning model includes an online network and a target network, wherein the online network is used to select an action and the target network is used to evaluate the Q value of the action;

[0081] According to the ε-greedy strategy, an online network is used to select actions, randomly selecting actions with ε probability, and selecting the action with the largest Q value with 1-ε probability. The ε value gradually decays during training;

[0082] The unmanned ship interacts with the environment according to the action with the largest Q value, and obtains an immediate reward and the next state. The immediate reward comprehensively considers the sailing distance, sailing time, energy consumption and obstacle avoidance factors;

[0083] The transfer samples consisting of the state, action, reward and next state are stored in a priority experience replay buffer, wherein the priority experience replay buffer determines its sampling priority according to the temporal difference error of the sample;

[0084] Sampling a batch of transfer samples from the priority experience replay buffer, calculating the temporal difference error of each transfer sample, using the target network to calculate the maximum Q value of the next state, and weighting the temporal difference error using the importance sampling weight to obtain a weighted mean square error loss function;

[0085] Based on the weighted mean square error loss function, the parameters of the online network are updated using a stochastic gradient descent algorithm, and the parameters of the online network are periodically copied to the target network; the above steps are repeated until the online network converges.

[0086] Exemplarily, this application uses the Deep Q-Network (DQN) algorithm to construct a deep reinforcement learning model for dynamic path planning of unmanned ships. The DQN algorithm is a value function-based reinforcement learning algorithm that approximates the state-action value function Q(s,a) through a deep neural network, enabling the agent to learn the optimal strategy in a continuous state space.

[0087] In the unmanned ship dynamic path planning problem, state s represents the current position coordinates (x, y) of the unmanned ship and the current ocean environment information, including current speed, wind speed, water depth, etc.; action a represents the next course and speed of the unmanned ship; reward r represents the degree of closeness of the unmanned ship to the optimal path after taking a certain action in the current state. The design of the reward function needs to comprehensively consider multiple factors such as sailing distance, sailing time, energy consumption, and obstacle avoidance.

[0088] The DQN algorithm uses a deep neural network to approximate the state-action value function Q(s,a). The input of the network is the state s, and the output is the Q value of each action taken in this state. During the training process, the network parameters are updated by minimizing the temporal difference error, so that the approximate Q function gradually converges to the optimal value function.

[0089] The DQN algorithm is a reinforcement learning algorithm that combines deep learning and Q learning. Its core idea is to use deep neural networks to approximate the state-action value function Q(s,a). Traditional Q learning algorithms use Q tables to store the value function of each state-action pair, but in a continuous state space, Q tables are difficult to store and converge. The DQN algorithm uses the powerful nonlinear fitting ability of deep neural networks to learn the optimal strategy directly from high-dimensional states without discretizing the state space in advance.

[0090] The training process of the DQN algorithm can be divided into the following steps:

[0091] State representation: The current position coordinates (x, y) of the unmanned ship and the ocean environment information (such as current speed, wind speed, water depth, etc.) are encoded into a real-valued vector as the input state s of the neural network.

[0092] Action selection: Based on the current state s, use the ε-greedy strategy to select action a. Randomly select actions with probability ε, and select the action with the largest Q value with probability 1-ε. The value of ε gradually decays during training, allowing the agent to achieve a balance between exploration and exploitation.

[0093] State transfer: The unmanned ship interacts with the environment according to the selected action a, and obtains the immediate reward r and the next state s'. The design of the reward function needs to comprehensively consider factors such as sailing distance, sailing time, energy consumption, and obstacle avoidance, and guide the unmanned ship to learn the optimal path planning strategy.

[0094] Experience storage: The transfer samples (s, a, r, s') are stored in the experience replay buffer D. The buffer is updated in a first-in-first-out manner and has a storage capacity of N.

[0095] Network training: Randomly extract a batch of transfer samples from the experience replay buffer D, and use the temporal difference learning algorithm to update the parameters of the Q network. For each sample (s, a, r, s'), calculate the temporal difference error:

[0096]

[0097] Among them, δ represents the temporal difference error, r represents the immediate reward, γ represents the discount factor, s, a, θ represent the current state, the current selected action and the parameters of the Q network respectively, s′, a′, θ - They represent the parameters of the next state, next action, and target network respectively.

[0098] Using the mean square error loss function, the error δ is gradient-dropped to update the parameters θ of the Q network. Every C steps, the parameters θ of the Q network are copied to the parameters θ of the target network. - Repeat the above steps until the Q network converges or reaches the preset number of training steps.

[0099] On this basis, the traditional DQN algorithm uses the same network to select actions and evaluate the Q value of actions when estimating the state-action value function Q(s,a), which may lead to overoptimistic Q value estimation and overfitting. The dual DQN algorithm solves this problem by introducing two independent Q networks, one for selecting actions and the other for evaluating the Q value of actions.

[0100] Specifically, the dual DQN algorithm maintains two Q networks with the same structure but different parameters, namely the online network Q(s,a;θ) and the target network Q(s,a;θ). - ). When selecting an action, an online network is used to select the action with the largest Q value:

[0101]

[0102] When calculating the target Q value, the target network is used to evaluate the maximum Q value of the next state:

[0103]

[0104] The target Q value y and the output Q(s,a;θ) of the online network are used to calculate the temporal difference error, and the parameters θ of the online network are updated by gradient descent. Every certain number of steps C, the parameters θ of the online network are copied to the parameters of the target network.

[0105] The advantage of the dual DQN algorithm is that the target network's parameter update frequency is lower than that of the online network, providing a relatively stable Q value estimate and reducing overfitting. At the same time, the decoupling of the online network and the target network also makes the algorithm more stable and less prone to divergence.

[0106] The traditional DQN algorithm uses uniform sampling to extract transfer samples from the experience replay buffer, which ignores the importance of different samples for learning. Prioritized Experience Replay (PER) is an improved experience replay mechanism that judges the importance of samples based on their temporal difference error. Samples with larger errors are given higher sampling priority.

[0107] Specifically, PER maintains a priority p for each transfer sample (s, a, r, s'), with the initial value being the maximum priority p_max. Each time a sample is sampled from the buffer, the i-th sample is selected with probability P(i):

[0108]

[0109] Among them, α is the priority coefficient, which controls the degree of deviation of the sampling distribution. When α=0, it degenerates into uniform sampling, and when α=1, it is completely sampled according to the priority. represents the sampling rate of the kth sample under the control of the priority coefficient, The sampling rate of the ith sample under the control of the priority coefficient.

[0110] PER accelerates the convergence speed of the Q network and improves the efficiency of sample utilization by giving priority to sampling important samples in the learning process. At the same time, the introduction of importance sampling weights also corrects the deviation caused by priority sampling and ensures the convergence of the algorithm.

[0111] S102. Based on the deep reinforcement learning model, the state value function is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states, and the optimal action strategy of the unmanned ship is determined according to the long-term rewards to obtain the local path planning result of the unmanned ship;

[0112] In an optional embodiment,

[0113] Based on the deep reinforcement learning model, the state value function is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states, and the optimal action strategy of the unmanned ship is determined according to the long-term rewards. The local path planning results of the unmanned ship include:

[0114] The current position coordinates, heading angle and ocean environment information within the local sensing range of the unmanned ship are encoded as a state vector, and the next heading change and speed change of the unmanned ship are encoded as an action vector;

[0115] Using a dueling deep Q network to evaluate the long-term rewards of the unmanned ship taking different actions in different states, the dueling deep Q network includes two independent neural networks for estimating the state value function and the advantage function respectively;

[0116] The state value function represents the expected cumulative reward of the unmanned ship in the current state, and the advantage function represents the advantage of selecting a certain action over other actions in the current state;

[0117] The output of the state-value function network is combined with the output of the advantage function network through the aggregation layer to obtain the final state-action value function, which is used to evaluate the long-term reward of taking an action in the current state;

[0118] During the training process, the unmanned ship updates the parameters of the dueling deep Q network using the temporal difference learning algorithm through interaction with the environment. The reward function comprehensively considers navigation safety, proximity to the target point, and smoothness of the path.

[0119] At the decision-making moment, the unmanned ship uses the trained dueling deep Q network to calculate the long-term reward of each action based on the current state, selects the action with the largest reward as the optimal decision, and obtains the next heading and speed;

[0120] Through a continuous decision-making process, the unmanned ship obtains a local path from its current position to the target position.

[0121] For example, in a dynamic environment, an unmanned ship needs to adjust its course and speed in real time based on the current state information to cope with the complex and changing marine environment. Traditional path planning methods based on optimization algorithms are difficult to adapt to the needs of dynamic environments, and deep reinforcement learning provides a new solution for local path planning of unmanned ships.

[0122] This application uses the dueling DQN algorithm to evaluate the long-term rewards of the unmanned ship taking different actions in different states, and then determine the optimal navigation strategy. Dueling DQN decomposes the state-action value function Q(s,a) into two parts: the state value function V(s) and the advantage function A(s,a):

[0123] Q(s,a)=V(s)+A(s,a);

[0124] Among them, the state value function V(s) represents the expected cumulative reward of the unmanned ship in state s, reflecting the inherent value of the state; the advantage function A(s,a) represents the advantage of choosing action a over other actions in state s, reflecting the relative value of the action.

[0125] Dueling DQN uses two independent neural networks to estimate the state-value function V(s) and the advantage function A(s,a) respectively. The state-value network inputs the state s and outputs the scalar V(s); the advantage network inputs the state s and outputs the vector A(s,a), where each element of the vector corresponds to the advantage value of an action. The outputs of the two networks are combined through an aggregation layer to obtain the final state-action value function Q(s,a):

[0126]

[0127] Among them, |A| is the size of the action space, 1 / |A|∑_a'A(s,a') is the mean of the advantage function, which is used to ensure the identity of Q(s,a).

[0128] In the local path planning problem of unmanned ships, the state s includes the current position coordinates (x, y) of the unmanned ship, the heading angle θ, and the ocean environment information within the local perception range, such as the location of obstacles, current speed, etc.; the action a is the next heading change Δθ and speed change Δv of the unmanned ship; the design of the reward r needs to comprehensively consider factors such as navigation safety, proximity to the target point, and smoothness of the path.

[0129] During the training process, the unmanned ship uses the dueling DQN algorithm to learn the state value function V(s) and advantage function A(s,a) through interaction with the environment. At each decision moment, the unmanned ship uses the trained dueling Q network to calculate the Q value of each action based on the current state s, and selects the action with the largest Q value as the optimal decision.

[0130] Through the continuous decision-making process, the unmanned ship can obtain a local path from the current position to the target position. In order to make the path smoother and more natural, a path curvature penalty term can be introduced into the reward function to encourage the unmanned ship to choose a path with less curvature.

[0131] In the testing phase, in order to improve the robustness of decision-making, the idea of ​​ensemble learning can be used to train multiple dueling Q networks, each of which is initialized with a different random seed. At each decision moment, the Q value outputs of multiple networks are averaged to obtain the final decision result. This ensemble decision-making method can reduce the accidental errors of a single network and improve the stability and reliability of path planning.

[0132] In summary, the dueling DQN algorithm based on the state value function provides a data-driven solution for the local path planning of unmanned ships. By learning the state value function to evaluate the long-term rewards under different states and selecting the action with the largest reward, the unmanned ship can adjust the course and speed in real time in a dynamic and unknown ocean environment to obtain a safe, efficient and smooth local navigation path.

[0133] S103. Based on a preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, the local path planning results are smoothly optimized to obtain a dynamic global path of the unmanned ship; according to the dynamic global path, the heading and speed of the unmanned ship are controlled so that the unmanned ship travels along the dynamic global path to the target position.

[0134] In an optional embodiment,

[0135] Based on the preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, the local path planning results are smoothly optimized to obtain the dynamic global path of the unmanned ship, including:

[0136] Using the local path planning result of the unmanned ship obtained by deep reinforcement learning as the input of the path optimization algorithm, the local path planning result is determined by the current position and target position of the unmanned ship;

[0137] Constructing a time-varying artificial potential field of the unmanned ship in the marine environment, wherein the time-varying artificial potential field is composed of a gravitational potential field of the target point, a repulsive potential field of the obstacle, and a velocity potential field considering the speed loss of the unmanned ship;

[0138] Introducing a time-varying gravitational coefficient and a time-varying repulsive coefficient into the potential field functions of the gravitational potential field and the repulsive potential field, respectively, wherein the time-varying gravitational coefficient and the time-varying repulsive coefficient are dynamically adjusted according to the magnitude of the ocean current velocity;

[0139] Establishing the motion equation of the unmanned ship in the time-varying artificial potential field, taking the waypoints in the local path planning result as the initial trajectory, using the gradient descent algorithm to iteratively optimize the total potential energy function, and obtaining a dynamic global path;

[0140] The dynamic global path is output to the motion control module of the unmanned ship to guide the autonomous decision-making and motion control of the unmanned ship.

[0141] For example, although the local path planning result of the unmanned ship obtained by the dueling DQN algorithm can make the best decision in real time according to the current state, the generated path may be redundant and not smooth because only a limited number of forward steps are considered. In order to further improve the quality of the path, it is necessary to optimize the local path on a larger spatiotemporal scale to generate a dynamically feasible global navigation path.

[0142] This application uses an improved artificial potential field method to perform post-processing optimization on the local path planning results of the unmanned ship. The traditional artificial potential field method has problems such as local minimum traps and oscillations, and is difficult to be directly applied to path planning in a dynamic ocean environment. To address this problem, this application introduces the concept of a time-varying artificial potential field, which enables the unmanned ship to adapt to the dynamic changes of the ocean environment by updating the potential field functions of obstacles and target points in real time.

[0143] First, the unmanned ship is regarded as a point mass, which is affected by the gravitational potential field of the target point and the repulsive potential field of the obstacle. In the traditional artificial potential field method, the gravitational potential field function of the target point is:

[0144]

[0145] Among them, U att (x) represents the gravitational potential field corresponding to the current position x of the unmanned ship, k att represents the gravitational coefficient, dgoal represents the Euclidean distance from the current position x of the unmanned ship to the target point;

[0146] The repulsive potential field function of the obstacle is:

[0147]

[0148] Among them, U rep (x) represents the repulsive potential field corresponding to the current position x of the unmanned ship, k rep represents the repulsion coefficient, d safe Indicates the safety distance threshold, d obs Indicates the shortest distance from the current position of the unmanned ship to the obstacle;

[0149] In order to adapt to the dynamic changes of the marine environment, this application improves the above potential field function. First, the time-varying gravity coefficient and time-varying repulsion coefficient are introduced to enable the unmanned ship to dynamically adjust the weights of gravity and repulsion according to the current sea conditions:

[0150] k att (t) = att0*exp(-λ att V current (t));

[0151] k rep (t) = rep0*exp(-λ rep V current ((t));

[0152] Among them, k att (t), k rep (t) represents the attraction coefficient and repulsion coefficient at time t, att0 and rep0 represent the initial attraction coefficient and initial repulsion coefficient, respectively. current (t) represents the ocean current velocity at time t.

[0153] Secondly, considering the dynamic constraints of the unmanned ship, the velocity loss term is introduced into the potential field function:

[0154]

[0155] Among them, U vel (x) represents the speed loss value of the unmanned ship at position x, k vel represents the velocity loss coefficient, v max represents the maximum speed of the unmanned ship, and v(x) represents the current speed of the unmanned ship at position x.

[0156] The speed loss term encourages the unmanned ship to maintain high speed and avoids jerky behaviors such as sharp turns and sudden acceleration. Taking the above factors into consideration, the total potential energy function of the unmanned ship at position x is:

[0157] U(x,t)=U att (x,t)+∑U rep (x,t)+U vel (x);

[0158] Among them, U(x,t) represents the total potential energy, U att (x, t) represents the gravitational potential field function corresponding to the current position x of the unmanned ship at time t, U rep (x, t) represents the repulsive potential field function corresponding to the current position x of the unmanned ship at time t, U vel (x) represents the speed loss value of the unmanned ship at position x.

[0159] At each decision moment, the first N waypoints are taken from the local path planning results obtained by the dueling DQN algorithm as the input of the optimization algorithm. With these N waypoints as the initial trajectory, the total potential energy function U(x,t) is iteratively optimized using the gradient descent method until convergence or the maximum number of iterations is reached. The optimized trajectory is used as the dynamic global path of the unmanned ship at the current moment and is output to the motion control module of the unmanned ship for execution.

[0160] In order to further improve the smoothness and feasibility of the path, path length penalty and curvature constraint can be added to the total potential energy function. The path length penalty makes the unmanned ship tend to choose a shorter navigation path; the curvature constraint limits the maximum curvature of the path so that the generated path meets the turning radius limit of the unmanned ship.

[0161] In summary, this application proposes a method for unmanned ship dynamic global path optimization based on the improved artificial potential field method. By introducing time-varying potential field functions and speed loss terms, the unmanned ship can adapt to the dynamic ocean environment and meet its own dynamic constraints. Based on the local path planning results obtained by the dueling DQN algorithm, a smooth, safe, and efficient global navigation path is further optimized to provide a reference for the autonomous decision-making and motion control of the unmanned ship.

[0162] Figure 2 FIG. 1 is a schematic diagram of a dynamic path planning system for an unmanned ship based on deep reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0163] The first unit is used to obtain the initial position, target position and marine environment information of the unmanned ship, wherein the marine environment information includes ocean current, wind speed, water depth and obstacle position; the initial position, target position and marine environment information of the unmanned ship are input into a pre-built deep reinforcement learning model, and the deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship through learning and training of sample data of dynamic path planning of the unmanned ship;

[0164] The second unit is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states using a state value function based on the deep reinforcement learning model, determine the optimal action strategy of the unmanned ship according to the long-term rewards, and obtain a local path planning result of the unmanned ship;

[0165] The third unit is used to smoothly optimize the local path planning results based on a preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, to obtain a dynamic global path of the unmanned ship; according to the dynamic global path, the heading and speed of the unmanned ship are controlled so that the unmanned ship travels to the target position along the dynamic global path.

[0166] According to a third aspect of the embodiments of the present invention,

[0167] An electronic device is provided, comprising:

[0168] processor;

[0169] a memory for storing processor-executable instructions;

[0170] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0171] A fourth aspect of the embodiments of the present invention is:

[0172] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0173] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The unmanned ship dynamic path planning method based on deep reinforcement learning is characterized by: include: Obtaining the initial position, target position and marine environment information of the unmanned ship, wherein the marine environment information includes ocean current, wind speed, water depth and obstacle position; The initial position, target position and ocean environment information of the unmanned ship are input into a pre-built deep reinforcement learning model, and the deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship by learning and training the sample data of the unmanned ship's dynamic path planning; Based on the deep reinforcement learning model, a state value function is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states, and the optimal action strategy of the unmanned ship is determined according to the long-term rewards to obtain the local path planning result of the unmanned ship; Based on a preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, the local path planning results are smoothly optimized to obtain a dynamic global path of the unmanned ship; according to the dynamic global path, the heading and speed of the unmanned ship are controlled so that the unmanned ship travels to the target position along the dynamic global path; Based on the preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, the local path planning results are smoothly optimized to obtain the dynamic global path of the unmanned ship, including: Using the local path planning result of the unmanned ship obtained by deep reinforcement learning as the input of the path optimization algorithm, the local path planning result is determined by the current position and target position of the unmanned ship; Constructing a time-varying artificial potential field of the unmanned ship in the marine environment, wherein the time-varying artificial potential field is composed of a gravitational potential field of the target point, a repulsive potential field of the obstacle, and a velocity potential field considering the speed loss of the unmanned ship; Introducing a time-varying gravitational coefficient and a time-varying repulsive coefficient into the potential field functions of the gravitational potential field and the repulsive potential field, respectively, wherein the time-varying gravitational coefficient and the time-varying repulsive coefficient are dynamically adjusted according to the magnitude of the ocean current velocity; Establishing the motion equation of the unmanned ship in the time-varying artificial potential field, taking the waypoints in the local path planning result as the initial trajectory, using the gradient descent algorithm to iteratively optimize the total potential energy function, and obtaining a dynamic global path; Output the dynamic global path to the motion control module of the unmanned ship to guide the autonomous decision-making and motion control of the unmanned ship; The gravitational potential field function of the target point is: Among them, U att (x) represents the gravitational potential field corresponding to the current position x of the unmanned ship, k att represents the gravitational coefficient, d goal Represents the Euclidean distance from the current position x of the unmanned ship to the target point; The repulsive potential field function of the obstacle is: Among them, U rep (x) represents the repulsive potential field corresponding to the current position x of the unmanned ship, k rep represents the repulsion coefficient, d safe Indicates the safety distance threshold, d obs Indicates the shortest distance from the current position of the unmanned ship to the obstacle; The time-varying attraction coefficient and time-varying repulsion coefficient are as follows: k att (t)=att0*exp(-λ att V current (t)); to rep (t)=rep0*exp(-λ rep IN current (t)); Among them, k att (t), k rep (t) represents the attraction coefficient and repulsion coefficient at time t, att0 and rep0 represent the initial attraction coefficient and initial repulsion coefficient, respectively. current (t) represents the ocean current velocity at time t; The speed loss is given by the following formula: Among them, U vel (x) represents the speed loss value of the unmanned ship at position x, k vel represents the velocity loss coefficient, v max represents the maximum speed of the unmanned ship, and v(x) represents the current speed of the unmanned ship at position x; The total potential energy function is given by the following formula: U(x,t)=U att (x,t)+∑U rep (x,t)+U vel (x); Among them, U(x,t) represents the total potential energy, U att (x, t) represents the gravitational potential field function corresponding to the current position x of the unmanned ship at time t, U rep (x, t) represents the repulsive potential field function corresponding to the current position x of the unmanned ship at time t, U vel (x) represents the speed loss value of the unmanned ship at position x.

2. The method according to claim 1, characterized in that: The initial position, target position and ocean environment information of the unmanned ship are input into a pre-built deep reinforcement learning model. The deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship through learning and training of the unmanned ship dynamic path planning sample data, including: Encoding the initial position of the unmanned ship and the ocean environment information into a state vector as the input of a deep reinforcement learning model, wherein the deep reinforcement learning model includes an online network and a target network, wherein the online network is used to select an action and the target network is used to evaluate the Q value of the action; According to the ε-greedy strategy, an online network is used to select actions, randomly selecting actions with ε probability, and selecting the action with the largest Q value with 1-ε probability. The ε value gradually decays during training; The unmanned ship interacts with the environment according to the action with the largest Q value, and obtains an immediate reward and the next state. The immediate reward comprehensively considers the sailing distance, sailing time, energy consumption and obstacle avoidance factors; The transfer samples consisting of the state, action, reward and next state are stored in a priority experience replay buffer, wherein the priority experience replay buffer determines its sampling priority according to the temporal difference error of the sample; Sampling a batch of transfer samples from the priority experience replay buffer, calculating the temporal difference error of each transfer sample, using the target network to calculate the maximum Q value of the next state, and weighting the temporal difference error using the importance sampling weight to obtain a weighted mean square error loss function; Based on the weighted mean square error loss function, the parameters of the online network are updated using a stochastic gradient descent algorithm, and the parameters of the online network are periodically copied to the target network; the above steps are repeated until the online network converges.

3. The method according to claim 2, characterized in that Calculating the maximum Q value of the next state using the target network includes: Among them, y represents the target Q value, r represents the immediate reward, γ represents the discount factor, a and θ represent the current selected action and the parameters of the Q network respectively, and s′ and θ - They represent the parameters of the next state and target network respectively.

4. The method according to claim 1, characterized in that: Based on the deep reinforcement learning model, the state value function is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states, and the optimal action strategy of the unmanned ship is determined according to the long-term rewards. The local path planning results of the unmanned ship include: The current position coordinates, heading angle and ocean environment information within the local sensing range of the unmanned ship are encoded as a state vector, and the next heading change and speed change of the unmanned ship are encoded as an action vector; Using a dueling deep Q network to evaluate the long-term rewards of the unmanned ship taking different actions in different states, the dueling deep Q network includes two independent neural networks for estimating the state value function and the advantage function respectively; The state value function represents the expected cumulative reward of the unmanned ship in the current state, and the advantage function represents the advantage of selecting a certain action over other actions in the current state; The output of the state-value function network is combined with the output of the advantage function network through the aggregation layer to obtain the final state-action value function, which is used to evaluate the long-term reward of taking an action in the current state; During the training process, the unmanned ship updates the parameters of the dueling deep Q network using the temporal difference learning algorithm through interaction with the environment; At the decision-making moment, the unmanned ship uses the trained dueling deep Q network to calculate the long-term reward of each action based on the current state, selects the action with the largest reward as the optimal decision, and obtains the next heading and speed; Through a continuous decision-making process, the unmanned ship obtains a local path from its current position to the target position.

5. The method according to claim 4, characterized in that The output of the state-value function network is combined with the output of the advantage function network through the aggregation layer to obtain the final state-action value function including: Among them, V(s) represents the state value function, A(s,a) represents the advantage function, |A| represents the size of the action space, s and a represent the current state and the currently selected action respectively, and a′ represents the next action.

6. An unmanned ship dynamic path planning system based on deep reinforcement learning, used to implement the method described in any one of claims 1 to 5, characterized in that: include: The first unit is used to obtain the initial position, target position and marine environment information of the unmanned ship, wherein the marine environment information includes ocean current, wind speed, water depth and obstacle position; The initial position, target position and ocean environment information of the unmanned ship are input into a pre-built deep reinforcement learning model, and the deep reinforcement learning model establishes a mapping relationship between the state, action and reward of the unmanned ship by learning and training the sample data of the unmanned ship's dynamic path planning; The second unit is used to evaluate the long-term rewards of the unmanned ship taking different actions in various states using a state value function based on the deep reinforcement learning model, determine the optimal action strategy of the unmanned ship according to the long-term rewards, and obtain a local path planning result of the unmanned ship; The third unit is used to smoothly optimize the local path planning results based on a preset path optimization algorithm, by considering the dynamic constraints of the unmanned ship and the dynamic changes of the marine environment, to obtain a dynamic global path of the unmanned ship; according to the dynamic global path, the heading and speed of the unmanned ship are controlled so that the unmanned ship travels to the target position along the dynamic global path.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images