Ship route automatic planning method based on deep reinforcement learning
By combining the experience of ship navigation experts and electronic nautical chart information with the DQN algorithm based on deep reinforcement learning, the route is automatically planned, which solves the problems of time-consuming, labor-intensive and unsafe navigation path planning in the existing technology and realizes safe and economical navigation path planning.
Patent Information
- Application Number
- CN202311510937.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-14
- Publication Date
- 2026-02-06
AI Technical Summary
Existing methods for planning ship routes are time-consuming and labor-intensive, prone to missing information, and difficult to ensure the safety and economy of navigation when considering the external environment and ship characteristics.
The DQN algorithm based on deep reinforcement learning is adopted, which combines the experience of ship navigation experts with electronic chart information. By extracting color features and using batch gradient descent, the route that avoids obstacles is automatically planned, and the Q-Learning algorithm is used to optimize the path decision.
It achieves safe and economical route planning, reduces the number of turns, improves the versatility and robustness of navigation, and matches the route design of ship navigation experts.
Smart Images
Figure CN121475211A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automatic ship route planning technology, specifically to an automatic ship route planning method based on deep reinforcement learning. Background Technology
[0002] With the continuous development of global informatization, the application of intelligent ships is becoming increasingly widespread. During ship navigation, in order to reduce fuel consumption and labor costs, ensure safe and reliable navigation, avoid many threats such as no-navigation zones, anchorages, shoals and reefs, and take into account factors such as ship draft and turning radius, designing a shortest and safest route is the key to ensuring safe and economical ship navigation.
[0003] Route design needs to consider both the external environment and the ship's own characteristics. Currently, there are two methods for route design: one is for navigation experts to manually draw routes using paper or electronic charts, based on external environmental factors such as wind speed, wind direction, tides, marine conditions, and weather, combined with the ship's own characteristics such as tonnage, draft, length, beam, height, and turning radius; the other is to use an electronic charter to generate a shortest route without obstacles by setting a starting point and ending point, and then correcting the shortest route based on the external environment, the ship's own characteristics, and electronic chart information. Both of these methods of route design are not only very time-consuming and labor-intensive, but also prone to overlooking information during the design process, leading to irreparable losses when the ship encounters unexpected situations during navigation.
[0004] In the study of automatic route planning for ships, graph theory algorithms, A* algorithms, and genetic algorithms are commonly used. Li Qing et al. compared the path planning of A* algorithms, genetic algorithms, and graph theory algorithms. The routes planned by A* algorithms and genetic algorithms were better than those planned by graph theory algorithms. However, A* algorithms cannot take into account external environment, ship characteristics, and electronic chart information. Genetic algorithms have slow convergence speed and are prone to getting trapped in local optima. The Deep Q-Network algorithm combines deep learning and reinforcement learning, cleverly integrating the perception and expression capabilities of deep learning with the decision-making and problem-solving capabilities of reinforcement learning. DQN is widely used in fields such as games, robot control, autonomous driving, and machine vision. Summary of the Invention
[0005] The purpose of this invention is to provide an automatic ship route planning method based on deep reinforcement learning. This method combines electronic chart information to avoid obstacles such as anchorages, restricted areas, and shipwrecks, and automatically plans routes using the DQN algorithm, ensuring the economy and safety of navigation. Simulation tests verify the feasibility of the DQN algorithm in fitting expert ship navigation routes. The results show that the planned route can navigate along the waterway with fewer turns, solving the problem of ordinary route planning algorithms that only seek the shortest route to avoid obstacles. It also has the advantages of strong versatility and robustness.
[0006] To achieve the above objectives, the present invention provides the following technical solution: an automatic ship route planning method based on deep reinforcement learning, the specific planning method of which is as follows:
[0007] (I) Framework for Ship Route Planning
[0008] To use DQN for ship route planning, you first need to input the experienced routes of a ship navigation expert and the nautical chart information for that route. The ultimate goal is to find a route that avoids all obstacles. The planning framework is as follows: Figure 1 ;
[0009] (II) Automatic Route Generation Method Based on DQN
[0010] A. DQN Algorithm
[0011] Based on the value function DQN algorithm and the deep Q network model, this model perceives and makes decisions on control tasks through images and maps. The Q network refers to the Q table generated by the Q-Learning reinforcement learning algorithm for decision-making.
[0012] The Q-Learning reinforcement learning algorithm can be used to make the target Q value a label, so that the calculated Q value and the target Q value form a loss function L(w)=E[(r+γmaxa'Q(s',a',w)-Q(s,a,w))2];
[0013] B. DQN based on experience-based routes
[0014] Directly using the experience routes of ship navigation experts and electronic nautical chart data for neural network training results in too much data and slow training speed. Therefore, the experience routes of ship navigation experts are fused with electronic nautical chart data images, and the images are processed into a single-channel encoding method through color feature extraction to shorten the training time of the neural network. When training the route planning, in order to prevent the training from only being able to adapt to familiar electronic nautical chart scenarios and not being able to adapt to unfamiliar electronic nautical chart scenarios and getting stuck in local optima, the batch gradient descent method is used to accelerate convergence, and a validation set is introduced. When the error rate of the validation set no longer decreases, the iteration stops.
[0015] (III) Experiment and Analysis
[0016] The DQN-based ship route planning method based on experience routes was adopted and the simulation was verified on a PC using Python 3.6.5. The Python modules used included PyQt5, tensorflow, pandas, numpy and matplotlib to simulate the electronic chart environment. The breadth-first search (BFS) algorithm was used to simulate the ship's experience route. The goal was to plan the route closest to the experience route through an intelligent agent.
[0017] The simulation platform first generates 10,000 10×10 matrix maps as training samples for DQN. Random obstacles are generated for each map, represented by solid squares. Outside the obstacle area, a path start point is randomly generated, represented by a square with an embedded rhombus outline. Outside the obstacle area and the start point, a path end point is randomly generated, represented by a square with an embedded circle outline. Shortest routes in eight directions are generated using BFS, simulating expert-experienced routes on real electronic nautical charts, represented by solid triangles. When the agent searches for a path, it represents a solid circle.
[0018] With path guidance, the agent searches for the path in each of 10,000 random maps. The convergence speed of the loss function is relatively close, but after about 1,000 training iterations, the loss function tends to 0. Therefore, during training, the number of training steps for each map is set to 1,000.
[0019] Since ξ-greedy is used to select the navigation direction, the impact of the random direction selection probability will be slightly different for the number of training steps and loss map for each map. When the random direction selection probability is 0.9, the greater the random direction selection probability, the greater the perturbation, but it is unrelated to the loss decay.
[0020] As a preferred embodiment, the environment of the ship route planning algorithm is based on a formal electronic nautical chart, which is rasterized in units of 300×300 pixels, and includes land, shipwrecks, and obstacles. After color feature extraction and contour recognition of the electronic nautical chart, the image is converted into a single-channel grayscale image to reduce training time. Here, 2×2 pixels with RGB values of (0, 255, 0) are the starting points of the route; 2×2 pixels with RGB values of (0, 0, 255) are the ending points of the route; and RGB values of (0, 0, 0) are the empirical routes. To avoid small-range oscillations in the path during reinforcement learning, the path width is set to 1 pixel.
[0021] As a preferred solution, the above method increases the number of data samplings, improves data utilization efficiency, and avoids the problems of low learning efficiency and high correlation between samples from continuous data. In actual training, the current parameters may determine the direction of the maximum reward action to the right. In this case, most samples come from the right side, and then the right side samples are trained again. This training may lead to a vicious cycle, with parameters showing local convergence or even divergence. By using the experience pool method, multiple states of data will be averaged, making the training process smooth and avoiding local convergence and divergence.
[0022] As a preferred option, in the real electronic chart system, 105 actual routes planned by experienced captains are used as training samples. After single-channel grayscale processing, they are used as training samples and directly input into the model. The samples are 300×300 pixel images, containing start point, end point, path, channel and obstacle data. The loss value converges in more than 400 steps and is more divergent compared with the simulation platform.
[0023] As a preferred embodiment, in the optimization problem of the neural network, the loss function is the difference between the label value and the network output. The goal is to minimize the loss function. To achieve this goal, a large amount of label data is required, and then the neural network parameters are updated by the error backpropagation gradient descent method.
[0024] As a preferred embodiment, since the ship route planning is implemented in electronic chart software, in order to distinguish colors, the chart display and image rendering of the electronic chart software are further developed to distinguish features such as routes, channels and turning points. The color feature extraction method is to find the color histogram difference between a certain pixel and its surrounding continuous pixels. Only when the difference is greater than a specified threshold can it be determined that the pixel has identifiable features with its surrounding pixels.
[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0026] 1. This invention combines electronic nautical chart information to avoid obstacles such as anchorages, restricted areas, and shipwrecks. It uses the DQN algorithm to automatically plan routes, ensuring the economy and safety of navigation. Simulation tests verify the feasibility of the DQN algorithm in fitting expert routes for ship navigation. The results show that the planned routes can navigate along waterways with fewer turns, solving the problem that ordinary route planning algorithms only seek the shortest route to avoid obstacles. Moreover, it has strong versatility and robustness. Attached Figure Description
[0027] Figure 1 This is a framework diagram for the DQN ship route planning of the present invention. Detailed Implementation
[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0029] Please see Figure 1 This invention relates to an automatic ship route planning method based on deep reinforcement learning, the specific planning method of which is as follows:
[0030] (I) Framework for Ship Route Planning
[0031] To use DQN for ship route planning, you first need to input the experienced routes of a ship navigation expert and the nautical chart information for that route. The ultimate goal is to find a route that avoids all obstacles. The planning framework is as follows: Figure 1 ;
[0032] (II) Automatic Route Generation Method Based on DQN
[0033] A. DQN Algorithm
[0034] Based on the value function DQN algorithm and the deep Q network model, this model perceives and makes decisions on control tasks through images and maps. The Q network refers to the Q table generated by the Q-Learning reinforcement learning algorithm for decision-making.
[0035] In the optimization problem of neural networks, the loss function is the difference between the label value and the network output. The goal is to minimize the loss function. To achieve this goal, a large amount of labeled data is needed, and then the neural network parameters are updated through the error backpropagation gradient descent method.
[0036] The Q-Learning reinforcement learning algorithm can be used to make the target Q value a label, so that the calculated Q value and the target Q value form a loss function L(w)=E[(r+γmaxa'Q(s',a',w)-Q(s,a,w))2];
[0037] B. DQN based on experience-based routes
[0038] Directly using the experience routes of ship navigation experts and electronic nautical chart data for neural network training results in too much data and slow training speed. Therefore, the experience routes of ship navigation experts are fused with electronic nautical chart data images, and the images are processed into a single-channel encoding method through color feature extraction to shorten the training time of the neural network. When training the route planning, in order to prevent the training from only being able to adapt to familiar electronic nautical chart scenarios and not being able to adapt to unfamiliar electronic nautical chart scenarios and getting stuck in local optima, the batch gradient descent method is used to accelerate convergence, and a validation set is introduced. When the error rate of the validation set no longer decreases, the iteration stops.
[0039] Color feature extraction initialization data
[0040] Since ship route planning is implemented in electronic chart software, in order to distinguish colors, secondary development is carried out on the chart display and image rendering of electronic chart software to distinguish features such as routes, channels and turning points. The color feature extraction method is to find the color histogram difference between a certain pixel and its surrounding continuous pixels. Only when the difference is greater than a specified threshold can it be determined that the pixel has identifiable features with its surrounding pixels.
[0041] (III) Experiment and Analysis
[0042] The DQN-based ship route planning method based on experience routes was adopted and the simulation was verified on a PC using Python 3.6.5. The Python modules used included PyQt5, tensorflow, pandas, numpy and matplotlib to simulate the electronic chart environment. The breadth-first search (BFS) algorithm was used to simulate the ship's experience route. The goal was to plan the route closest to the experience route through an intelligent agent.
[0043] The simulation platform first generates 10,000 10×10 matrix maps as training samples for DQN. Random obstacles are generated for each map, represented by solid squares. Outside the obstacle area, a path start point is randomly generated, represented by a square with an embedded rhombus outline. Outside the obstacle area and the start point, a path end point is randomly generated, represented by a square with an embedded circle outline. Shortest routes in eight directions are generated using BFS, simulating expert-experienced routes on real electronic nautical charts, represented by solid triangles. When the agent searches for a path, it represents a solid circle.
[0044] With path guidance, the agent searches for the path in each of 10,000 random maps. The convergence speed of the loss function is relatively close, but after about 1,000 training iterations, the loss function tends to 0. Therefore, during training, the number of training steps for each map is set to 1,000.
[0045] Since ξ-greedy is used to select the navigation direction, the impact of the random direction selection probability will be slightly different for the number of training steps and loss map for each map. When the random direction selection probability is 0.9, the greater the random direction selection probability, the greater the perturbation, but it is unrelated to the loss decay.
[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.
Claims
1. A method for automatic ship route planning based on deep reinforcement learning, characterized in that: The specific planning method is as follows: (I) Framework for Ship Route Planning To use DQN for ship route planning, you need to first input the experienced route of a ship navigation expert and the nautical chart information of that route. The ultimate goal is to find a route that avoids all obstacles. Its planning framework is shown in Figure 1. (II) Automatic Route Generation Method Based on DQN A. DQN Algorithm Based on the value function DQN algorithm and the deep Q network model, this model perceives and makes decisions on control tasks through images and maps. The Q network refers to the Q table generated by the Q-Learning reinforcement learning algorithm for decision-making. The Q-Learning reinforcement learning algorithm can be used to make the target Q value a label, so that the calculated Q value and the target Q value form a loss function L(w)=E[(r+γmaxa'Q(s',a',w)-Q(s,a,w))2]; B. DQN based on experience-based routes Directly using the experience routes of ship navigation experts and electronic nautical chart data for neural network training results in too much data and slow training speed. Therefore, the experience routes of ship navigation experts are fused with electronic nautical chart data images, and the images are processed into a single-channel encoding method through color feature extraction to shorten the training time of the neural network. When training the route planning, in order to prevent the training from only being able to adapt to familiar electronic nautical chart scenarios and not being able to adapt to unfamiliar electronic nautical chart scenarios and getting stuck in local optima, the batch gradient descent method is used to accelerate convergence, and a validation set is introduced. When the error rate of the validation set no longer decreases, the iteration stops. (III) Experiment and Analysis The DQN-based ship route planning method based on experience routes was adopted and the simulation was verified on a PC using Python 3.6.
5. The Python modules used included PyQt5, tensorflow, pandas, numpy and matplotlib to simulate the electronic chart environment. The breadth-first search (BFS) algorithm was used to simulate the ship's experience route. The goal was to plan the route closest to the experience route through an intelligent agent. The simulation platform first generates 10,000 10×10 matrix maps as training samples for DQN. Random obstacles are generated for each map, represented by solid squares. Outside the obstacle area, a path start point is randomly generated, represented by a square with an embedded rhombus outline. Outside the obstacle area and the start point, a path end point is randomly generated, represented by a square with an embedded circle outline. Shortest routes in eight directions are generated using BFS, simulating expert-experienced routes on real electronic nautical charts, represented by solid triangles. When the agent searches for a path, it represents a solid circle. With path guidance, the agent searches for the path in each of 10,000 random maps. The convergence speed of the loss function is relatively close, but after about 1,000 training iterations, the loss function tends to 0. Therefore, during training, the number of training steps for each map is set to 1,000. Since ξ-greedy is used to select the navigation direction, the impact of the random direction selection probability will be slightly different for the number of training steps and loss map for each map. When the random direction selection probability is 0.9, the greater the random direction selection probability, the greater the perturbation, but it is unrelated to the loss decay.
2. The automatic ship route planning method based on deep reinforcement learning according to claim 1, characterized in that: The ship route planning algorithm is based on a formal electronic nautical chart, rasterized in 300×300 pixels, containing land, shipwrecks, and obstacles. After color feature extraction and contour recognition of the electronic nautical chart, the image is converted into a single-channel grayscale image to reduce training time. Specifically, 2×2 pixels with RGB values of (0, 255, 0) represent the starting point of the route; 2×2 pixels with RGB values of (0, 0, 255) represent the ending point of the route; and RGB values of (0, 0, 0) represent the empirical route. To avoid small-range oscillations in the path during reinforcement learning, the path width is set to 1 pixel.
3. The automatic ship route planning method based on deep reinforcement learning according to claim 1, characterized in that: The above-mentioned method increases the number of data samplings, improves data utilization efficiency, and avoids the problems of low efficiency and high correlation between samples from continuous data. In actual training, the current parameters may determine the direction of the maximum reward action to the right. In this case, most samples come from the right side, and then the right side samples are trained again. This training may lead to a vicious cycle, with parameters showing local convergence or even divergence. By using the experience pool method, multiple states of data are averaged, making the training process smooth and avoiding local convergence and divergence.
4. The automatic ship route planning method based on deep reinforcement learning according to claim 1, characterized in that: In the real electronic chart system, 105 actual routes planned by experienced captains were used as training samples. After single-channel grayscale processing, these samples were directly input into the model. The samples were 300×300 pixel images containing start point, end point, path, channel, and obstacle data. The loss value converged in more than 400 steps and was more divergent compared to the simulation platform.
5. The automatic ship route planning method based on deep reinforcement learning according to claim 1, characterized in that: In the optimization problem of neural networks, the loss function is the difference between the label value and the network output. The goal is to minimize the loss function. To achieve this goal, a large amount of labeled data is needed, and then the neural network parameters are updated through the error backpropagation gradient descent method.
6. The automatic ship route planning method based on deep reinforcement learning according to claim 1, characterized in that: Since ship route planning is implemented in electronic chart software, in order to distinguish colors, secondary development is carried out on the chart display and image rendering of the electronic chart software to distinguish features such as routes, channels and turning points. The color feature extraction method is to find the color histogram difference between a certain pixel and its surrounding continuous pixels. Only when the difference is greater than a specified threshold can it be determined that the pixel has identifiable features with its surrounding pixels.