A large ship path planning method integrating DQN and artificial potential field

Through the large-scale ship path planning method that integrates DQN and artificial potential fields, the problem of poor feasibility of large-scale ship path planning in the existing technology is solved, and more efficient and smoother path planning is achieved, which is suitable for the actual navigation needs of large ships.

CN118534913BActive Publication Date: 2024-11-29NAT UNIV OF DEFENSE TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411006862.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-29
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The existing path planning methods have poor feasibility when applying large ships, and cannot effectively realize trajectories with large heading changes.

Method used

Using a large ship path planning method that integrates DQN and artificial potential fields, by obtaining navigation environment information of navigation areas, dividing allowable navigation areas and navigation restricted areas, building a combined force field model based on artificial potential fields and a reward function based on DQN algorithm, and performing depth Reinforcement learning search processing, obtain path planning data, and perform smoothing processing through ship simulation model.

Benefits of technology

It improves the feasibility of path planning for large ships, enhances the purpose and efficiency of path search, takes into account the large inertia and time delay of large ships, and the generated path is smoother and conforms to the maneuvering characteristics of large ships.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118534913B_ABST
    Figure CN118534913B_ABST
Patent Text Reader

Abstract

The present application discloses a large ship path planning method integrating DQN and artificial potential field, including: obtaining navigation environment information of the navigation area, dividing the navigation area into a permitted navigation area and a navigation restricted area according to the navigation environment information, and obtaining grid map data of the navigation area; constructing a resultant field model based on the artificial potential field according to the grid map data of the navigation area; constructing a reward function based on the DQN algorithm according to the resultant field model; performing deep reinforcement learning search processing of the ship's navigation path according to the reward function to obtain first path planning data; constructing a ship simulation model of the corresponding ship, performing ship steady turn simulation on the ship simulation model, and obtaining ship simulation navigation trajectory data; performing smoothness processing on the first path planning data according to the ship simulation navigation trajectory data to obtain second path planning data. The feasibility of large ship path planning can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of ship transportation, and in particular to a large ship path planning method integrating DQN and artificial potential field. Background Art

[0002] Path planning research is an important part of the development of unmanned ship intelligent navigation system, and is of great significance to promoting the safety and efficiency of marine transportation. However, this research faces problems such as complex ship navigation environment and high safety requirements. Some methods have been proposed in related technologies to solve the above problems, including bionic algorithms based on biological populations, A* algorithms based on graph theory, artificial potential field methods based on physical models, and intelligent algorithms based on data-driven.

[0003] However, the above solutions are basically path planning for small ships, while ignoring large ships. Compared with small ships, large ships have poor maneuverability and cannot achieve trajectories with large heading changes.

[0004] In summary, the existing path planning methods have the problem of poor feasibility when applied to path planning of large ships. Summary of the invention

[0005] This application proposes a large ship path planning method that integrates DQN and artificial potential field, which can improve the feasibility of large ship path planning.

[0006] A large ship path planning method integrating DQN and artificial potential field according to an embodiment of the present application includes the following steps:

[0007] Acquire navigation environment information of a navigation area, where the navigation area is the area between a starting point and a target point of the ship;

[0008] Dividing the navigation area into a permitted navigation area and a restricted navigation area according to the navigation environment information, and obtaining grid map data of the navigation area;

[0009] Constructing a resultant force field model based on an artificial potential field according to grid map data of the navigation area, wherein the resultant force field model includes a gravitational field sub-model and a repulsive field sub-model;

[0010] Constructing a reward function based on the DQN algorithm according to the resultant force field model;

[0011] Performing deep reinforcement learning search processing on the ship's navigation path according to the reward function to obtain first path planning data;

[0012] Constructing a ship simulation model corresponding to the ship, performing a ship steady turn simulation on the ship simulation model, and obtaining ship simulation navigation track data;

[0013] The first path planning data is smoothed according to the simulated navigation trajectory data of the ship to obtain second path planning data, where the second path planning data is used to instruct the ship to navigate from the starting point to the target point.

[0014] According to some embodiments of the present application, in the step of constructing a resultant field model based on an artificial potential field according to the grid map data of the navigation area, wherein the resultant field model includes a gravitational field sub-model and a repulsive field sub-model, the expression of the gravitational field sub-model is:

[0015] ;

[0016] in, is the gravitational coefficient, Represents the Euclidean distance from the ship to the target point.

[0017] According to some embodiments of the present application, in the step of constructing a resultant field model based on an artificial potential field according to the grid map data of the navigation area, wherein the resultant field model includes a gravitational field sub-model and a repulsive field sub-model, the expression of the repulsive field sub-model is:

[0018] ;

[0019] in, is the repulsion coefficient, is the Euclidean distance from the ship to the restricted navigation zone, It is the scope of influence of the restricted navigation zone.

[0020] According to some embodiments of the present application, in the step of constructing a reward function based on the DQN algorithm according to the resultant force field model, the expression of the reward function is:

[0021] ;

[0022] in, is the gravitational field submodel, It is a repulsive field model.

[0023] According to some embodiments of the present application, the deep reinforcement learning search process of the ship navigation path according to the reward function to obtain the first path planning data includes:

[0024] Constructing an initialized neural network model according to the reward function;

[0025] Fitting the action value function according to the initialized neural network model to obtain environmental observation sequence data, and storing the environmental observation sequence data in an experience pool;

[0026] The initialized neural network model is trained through the experience pool to obtain the trained main network model;

[0027] Updating the fitted action-value function according to the main network model to obtain an updated action-value function;

[0028] A search process is performed according to the updated action value function to obtain first path planning data.

[0029] According to some embodiments of the present application, the training of the initialized neural network model through the experience pool to obtain the trained main network model also includes:

[0030] A target network model based on the main network model is constructed, and the target network model copies the network parameters of the main network model according to the target time interval.

[0031] According to some embodiments of the present application, in the step of performing search processing based on the updated action-value function to obtain the first path planning data, the updated action-value function is searched based on a time-varying greedy strategy to obtain the first path planning data.

[0032] According to some embodiments of the present application, the step of constructing a ship simulation model corresponding to the ship, performing a ship steady turn simulation on the ship simulation model, and obtaining ship simulation navigation track data includes:

[0033] Construct a ship simulation model of the corresponding ship;

[0034] obtaining a steering gear servo system associated with the turning performance of the vessel;

[0035] According to the ship simulation model and the steering gear servo system, the ship simulation model is simulated for steady turning to obtain the position information of the ship when turning.

[0036] The position information of the ship when turning is expanded according to the symmetry algorithm to obtain the ship simulation navigation track data.

[0037] According to some embodiments of the present application, the ship simulation model is a nonlinear Nomoto model.

[0038] According to some embodiments of the present application, the step of performing expansion processing on the position information of the ship when turning according to the symmetry algorithm includes:

[0039] Obtaining course change trajectory information according to the position information of the ship when turning, wherein the course change trajectory information is trajectory information in which the course of the ship changes by 90° during navigation;

[0040] Performing expansion processing on the course change trajectory information according to a symmetry algorithm to obtain expanded course change trajectory information;

[0041] The expanded course change trajectory information is translated to the right-angle turn to obtain the ship's simulated navigation trajectory data.

[0042] A large ship path planning method integrating DQN and artificial potential field according to an embodiment of the present application has at least the following beneficial effects:

[0043] This implementation method first obtains the navigation environment information of the navigation area; then divides the navigation area into a permitted navigation area and a navigation restricted area according to the navigation environment information, and obtains the grid map data of the navigation area; constructs a resultant field model based on an artificial potential field according to the grid map data of the navigation area; constructs a reward function based on the DQN algorithm according to the resultant field model; performs deep reinforcement learning search processing of the ship's navigation path according to the reward function to obtain the first path planning data; constructs a ship simulation model corresponding to the ship, performs a ship steady turn simulation on the ship simulation model, and obtains the ship simulation navigation trajectory data; performs smoothness processing on the first path planning data according to the ship simulation navigation trajectory data to obtain the second path planning data, and the second path planning data is used to indicate that the ship sails from the starting point to the target point. The present invention uses the artificial potential field method to improve the reward function in the DQN algorithm, which not only makes the reward function continuous, but also makes the path search more purposeful, thereby improving the search efficiency. In addition, the first path planning data is smoothed according to the ship simulation navigation trajectory data, taking into account the large inertia and time lag of large ships, which can improve the feasibility of large ship path planning.

[0044] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present application is further described below with reference to the accompanying drawings and embodiments, wherein:

[0046] Figure 1 This is a flow chart of a large ship path planning method integrating DQN and artificial potential field in an embodiment of the present application;

[0047] Figure 2 This is an architecture diagram of the deep reinforcement learning algorithm in the embodiment of this application;

[0048] Figure 3 This is a framework diagram of a ship turning simulation experiment in an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the principle of expanding the ship steering information in the embodiment of the present application;

[0050] Figure 5aThis is a schematic diagram of the Q value change of the DQN algorithm in the experimental example;

[0051] Figure 5b A schematic diagram of the Q value change of the method adopted in the embodiment of the present application in the experimental example;

[0052] Figure 6a This is a diagram showing the number of successful obstacle avoidance experiments of the DQN algorithm in the experimental example;

[0053] Figure 6b A schematic diagram of the number of successful obstacle avoidance experiments using the method used in the embodiment of the present application in the experimental example;

[0054] Figure 7 This is a schematic diagram of the results of the ship steady turning simulation test in the experimental example;

[0055] Figure 8 Schematic diagram of the final planned smooth trajectory in the experimental example. DETAILED DESCRIPTION

[0056] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0057] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0058] In the description of this application, "a plurality" means more than two. If there is a description of "first" or "second", it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0059] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0060] Reference Figure 1 As shown, a large ship path planning method integrating DQN and artificial potential field includes the following steps:

[0061] S101, obtaining navigation environment information of a navigation area, where the navigation area is the area between a starting point and a target point of a ship;

[0062] Specifically, the above-mentioned navigation environment information refers to obstacle information obtained through environmental perception technology, such as islands, reefs, other ships, etc., by perceiving the environment based on radar, AIS, electronic nautical charts and other equipment carried by the ship, thereby obtaining navigation environment information within the navigation area.

[0063] S102, dividing the navigation area into a permitted navigation area and a restricted navigation area according to the navigation environment information, and obtaining grid map data of the navigation area;

[0064] Specifically, the above-mentioned restricted navigation area refers to an area in the navigation area where obstacles exist. The restricted navigation area must be avoided in ship path planning, and all areas in the navigation area except the restricted navigation area are regarded as permitted navigation areas.

[0065] S103, constructing a resultant force field model based on an artificial potential field according to the grid map data of the navigation area, wherein the resultant force field model includes a gravitational field sub-model and a repulsive field sub-model;

[0066] It should be noted that the artificial potential field achieves the purpose of enabling the controlled object to avoid obstacles and eventually reach the target point by imitating the attraction and repulsion between electric charges. Therefore, the key to the artificial potential field is how to construct the gravitational field and the repulsive field, so that the farther the controlled object is from the obstacle, the smaller the repulsive force it receives; the farther it is from the target point, the greater the gravitational force it receives, and finally reaches the target point under the action of the combined force.

[0067] Assume that in a certain plane space, a ship is in a gravitational field and repulsive field Under the dual effect of , starting from the starting point. The spatial position coordinates of the starting point and the target point in the inertial coordinate system are , The combined potential field on this ship is as follows:

[0068] ;

[0069] In the above formula X Indicates the coordinate position of the ship during navigation The gravitational field function of the ship, that is, the expression of the gravitational field sub-model is:

[0070] ;

[0071] in, is the gravitational coefficient, Represents the Euclidean distance from the ship to the target point.

[0072] It should be noted that the repulsion field function is a quadratic function, and the independent variable is the inverse of the Euler distance from the ship to the obstacle. Therefore, a small movement of the ship will cause a drastic change in the repulsion force and thus aggravate the buffeting phenomenon. Therefore, this application selects an exponential function as the repulsion field function and introduces the Euler distance from the target point to the ship. , so the repulsive field function, that is, the expression of the repulsive field sub-model is:

[0073] ;

[0074] in, is the repulsion coefficient, is the Euclidean distance from the ship to the restricted navigation zone, It is the scope of influence of the restricted navigation zone.

[0075] S104. Construct a reward function based on the DQN (Deep Q Network) algorithm according to the combined force field model;

[0076] It should be noted that the reward function is a core component in reinforcement learning, which defines the immediate reward obtained by the agent after performing a specific action. The resultant force field model based on the artificial potential field constructed in step S103 is to replace the original piecewise reward function in the DQN algorithm with the resultant force field acting on the ship in the artificial potential field method.

[0077] Specifically, the expression of the reward function in this application is:

[0078] ;

[0079] in, is the gravitational field submodel, It is a repulsive field model.

[0080] and They represent the gravitational field sub-model and the repulsive field sub-model, that is, the gravitational field and repulsive field functions of the artificial potential field method. The reward function used in this application ignores the direction of the force and simply uses the magnitude of the force as the reward or penalty. The closer the ship's position is to the target point, the smaller the gravity and the larger the reward value r; the closer it is to the obstacle, the greater the repulsive force and the larger the reward value r The smaller the value, the smaller the reward function. The reward function incorporating the artificial potential field method has continuity, which makes the search process of deep reinforcement learning more purposeful, thereby improving learning efficiency.

[0081] S105, performing deep reinforcement learning search processing on the ship's navigation path according to the reward function to obtain first path planning data;

[0082] The specific steps of the above step S105 are:

[0083] Build an initialized neural network model based on the reward function;

[0084] Fitting the action value function according to the initialized neural network model, obtaining the environmental observation sequence data, and storing the environmental observation sequence data into the experience pool;

[0085] The initialized neural network model is trained through the experience pool to obtain the trained main network model;

[0086] Update the fitted action-value function according to the main network model to obtain an updated action-value function;

[0087] A search process is performed according to the updated action value function to obtain the first path planning data.

[0088] It should be noted that the above deep reinforcement learning search process is processed by the deep reinforcement learning algorithm. The deep reinforcement learning algorithm is divided into two periods: the observation period T obs and training period T train .

[0089] The first phase is the observation phase, which is used to generate data for the experience pool. During the observation phase, the action value function, i.e., Q value, is fitted using the initialized neural network. After obtaining a series of environmental observation sequence data, the data is stored in the experience pool. It should be noted that the action value function (Q value) is used to estimate the expected return (Expected Return) of taking a specific action (Action) in a specific state (State). It represents the expected value of the long-term cumulative reward that the agent can obtain by taking action a in state s and following a certain strategy π.

[0090] Then it enters the training phase, during which the main network is actually evaluated and trained, and the Q value is continuously updated to approach the optimal Q value.

[0091] It should be noted that in the present application, the updated action-value function is searched and processed based on the time-varying greedy strategy to obtain the first path planning data.

[0092] It should be noted that the initialized neural network model is trained through the experience pool to obtain the trained main network model, which also includes:

[0093] A target network model based on the main network model is constructed, and the target network model copies the network parameters of the main network model according to the target time interval.

[0094] The purpose of the target network model is to renewThe network parameters of the main network are copied from the target network model to the target network model, and the target network model is used to process the deviation in the timing difference. The structure of the target network model is the same as that of the target network model, and the parameters of the main network are copied at regular intervals. This processing not only enhances the stability of the algorithm, but also builds a label for the main network to facilitate its parameter update.

[0095] The specific principles of the deep reinforcement learning algorithm in the embodiment of this application are described in detail below:

[0096] refer to Figure 2 As shown, traditional reinforcement learning uses table storage Q Values ​​are indexed by state and action. However, in practical applications, the state space and action space may be continuous and cannot be stored in a table. Therefore, this application combines deep learning with reinforcement learning and uses neural networks to obtain approximate Q value, and by updating the network parameters make Close to reality Q value. Figure 2 The APF (Artificial Potential Field) environment module replaces the piecewise reward function in the original DQN algorithm with the resultant force field acting on the ship in the artificial potential field method.

[0097] The structural analysis of each part of the framework is as follows:

[0098] 1) Convolutional neural network is based on maximum likelihood estimation to approximate the real Q The premise is that the training samples are independent and identically distributed. However, the data obtained through interaction during the learning process has a certain correlation. In order to solve this problem and improve the utilization rate of samples and the stability of training, an experience replay pool is introduced. The pool can continuously store state, action, and reward data, and randomly extract data from it during the learning process.

[0099] 2) The main network is a convolutional neural network used to fit the state-action value. The input of the network is the state of the agent at a certain moment and the action selected, and the output is the corresponding Q value.

[0100] 3) The target network is used to process the deviation in the time series difference. The structure of this network is the same as that of the main network, and the parameters of the main network are copied at regular intervals. This processing not only enhances the stability of the algorithm, but also builds a label for the main network to facilitate its parameter update.

[0101] 4) The loss function is shown below.

[0102] ;

[0103] The parameters of the main network are updated as follows.

[0104] ;

[0105] In the formula is the gradient of the main network. In fact, it is the goal of the algorithm process Q Value, updated based on the time series difference method Q Value, as shown below:

[0106] ;

[0107] 5) Adoption The greedy strategy is to search. The probability is evenly distributed to each action in the action set. For the optimal action, add At the same time, by changing ε The value allows the random search in the later stage of training to reduce the interference of the optimal strategy. The specific strategy expression is as follows:

[0108] ;

[0109] ;

[0110] is the round of algorithm training, for In the initial stage, the algorithm searches for paths with a 100% probability. After 100 trainings, it continues to search with a certain degree of randomness. ε The value is time-varying, which minimizes the computational cost and makes the final strategy more stable.

[0111] S106, constructing a ship simulation model corresponding to the ship, performing a ship steady turn simulation on the ship simulation model, and obtaining ship simulation navigation track data;

[0112] It should be noted that the first path planning data obtained in step S105 is only a route path planning based on the grid map data, without considering the steering gear and turning performance limitations of large ships. Therefore, there may be unreasonable aspects when applied to large ships.

[0113] The specific steps of step S106 are:

[0114] Construct a ship simulation model of the corresponding ship;

[0115] Acquiring the steering gear servo system associated with the turning performance of the vessel;

[0116] According to the ship simulation model and the steering gear servo system, the ship simulation model is simulated for steady turning to obtain the position information of the ship when turning.

[0117] The position information of the ship when turning is expanded and processed according to the symmetry algorithm to obtain the ship's simulated navigation trajectory data.

[0118] Specifically, the ship simulation model in this application adopts the nonlinear Nomoto model, which is expressed as follows:

[0119] ;

[0120] Among them, α, β, K.T They are the ship maneuverability index, which is determined by many factors such as ship structural parameters and navigation conditions; The actual heading, is the angular velocity, is the angular acceleration; is the rudder angle; Since the present application aims to make the planned path smoother according to the turning performance of the ship, the external interference is ignored, that is, Δ is 0. The ship maneuvering coefficient can be obtained through the ship type parameters of the specific ship.

[0121] The above-mentioned steady turning test of a ship refers to a turning motion test conducted by a ship under a certain initial speed and rudder angle. In this experiment, the ship will perform continuous circular motion around a fixed point until it reaches a stable state, at which time the ship's turning radius, speed, heading and other parameters no longer change. This state is called a steady turning state. Steady turning tests are usually used to evaluate the maneuverability and turning performance of a ship, such as evaluating the maneuverability of a ship by measuring parameters such as the ship's turning diameter, turning period, and maximum heel angle.

[0122] The nonlinear Nomoto model is used to build a simulation platform on Simlink to obtain the ship steady turning experimental trajectory. The simulation structure is as follows: Figure 3 As shown. In order to make the simulation fit the navigation practice, the steering gear servo system is added to the simulation structure. The servo system includes the rudder angle saturation limit, the rudder angle steering speed limit and the first-order inertial system 1 / (6s+1). The role of this inertial system is to simulate the transition process of the ship's rudder angle response to improve the accuracy of the system simulation.

[0123] It should be noted that the output of the nonlinear Nomoto model is the angular velocity of the heading, which can be integrated to obtain the heading angle However, the actual position information of the ship when turning is needed to perform smooth processing on the first path planning data. The specific conversion formula is as follows:

[0124] ;

[0125] In the above formula, (x, y) is the position of the ship. and They represent the components of the velocity on the x-axis and y-axis respectively; U is the speed of the ship; The ship's heading.

[0126] The position information of the ship when turning is expanded and processed according to the symmetry algorithm, including:

[0127] The course change trajectory information is obtained according to the position information of the ship when turning, and the course change trajectory information is the trajectory information of the course change of 90° during the navigation process of the ship;

[0128] Expanding the course change trajectory information according to a symmetry algorithm to obtain expanded course change trajectory information;

[0129] The expanded course change trajectory information is translated to the right-angle turn to obtain the ship's simulated navigation trajectory data.

[0130] It should be noted that among the position information of the ship when turning obtained through the ship steady turning simulation test, only the trajectory of the ship sailing from the starting point and changing its heading by 90° contains the time delay effect of the steering gear response. Therefore, only this section of trajectory information is retained and expanded using the symmetry algorithm to obtain the trajectory of 90 degrees to 180 degrees, such as Figure 4 As shown in the figure. The trajectory of the first quadrant is the position information (x, y) obtained through the simulation experiment. The position information of the other three quadrants is obtained through the symmetry of the rectangular coordinate system, that is, the positive and negative signs of the x and y values ​​are changed based on the information (x, y) in the first quadrant. Finally, the trajectory information obtained by the expansion is translated to the right-angle turn to obtain the simulated navigation trajectory data of the ship.

[0131] S107, performing smoothing processing on the first path planning data according to the ship simulation navigation trajectory data to obtain second path planning data, where the second path planning data is used to instruct the ship to navigate from the starting point to the target point.

[0132] It should be understood that after obtaining the ship simulation navigation trajectory data taking into account the response of the large ship's steering gear, the first path planning data is processed using this trajectory, and the second path planning data obtained is a theoretically optimal route path planning with a smoother turning path and in line with the maneuverability of large ships.

[0133] This implementation method first obtains the navigation environment information of the navigation area; then divides the navigation area into a permitted navigation area and a navigation restricted area according to the navigation environment information, and obtains the grid map data of the navigation area; constructs a resultant field model based on an artificial potential field according to the grid map data of the navigation area; constructs a reward function based on the DQN algorithm according to the resultant field model; performs deep reinforcement learning search processing of the ship's navigation path according to the reward function to obtain the first path planning data; constructs a ship simulation model corresponding to the ship, performs a ship steady turn simulation on the ship simulation model, and obtains the ship simulation navigation trajectory data; performs smoothness processing on the first path planning data according to the ship simulation navigation trajectory data to obtain the second path planning data, and the second path planning data is used to indicate that the ship sails from the starting point to the target point. The present invention uses the artificial potential field method to improve the reward function in the DQN algorithm, which not only makes the reward function continuous, but also makes the path search more purposeful, thereby improving the search efficiency. In addition, the first path planning data is smoothed according to the ship simulation navigation trajectory data, taking into account the large inertia and time lag of large ships, which can improve the feasibility of large ship path planning.

[0134] The effects of the present invention are described in detail below with a specific large ship path planning experiment example:

[0135] In this example, the navigation area of ​​the ship is set as a calm sea area of ​​3000m×3000m, and it is divided into 30×30 grids. The hidden layer of the neural network is set to 2 layers and has 60 neurons. The specific neural network parameters are shown in Table 1.

[0136]

[0137] This application uses a conventional single DQN algorithm as a comparative example to perform obstacle avoidance in the same sea area to verify the effectiveness of the method of this application in avoiding obstacles in complex sea areas. From the perspective of navigation practice, the minimum distance from the planned path to the obstacle is used. , the length of the planned path L And the number of waypoints N To evaluate the safety, economy and feasibility of the planned path.

[0138] like Figure 5a , Figure 6a As shown in the figure, the DQN algorithm successfully randomly searched two paths during the observation period. The learning effect was obvious in the early stage after entering the training period, but after 300 rounds, no path to the target point was successfully searched. This is because a large disturbance occurred during the random search process around 300 rounds, but the subsequent learning did not eliminate this disturbance. Therefore, the strategy learned after the training is completed cannot correctly guide the ship to the target point.

[0139] like Figure 5b and Figure 6b As shown, after the path optimization method adopted in steps S101-S105 of the present application enters the training period, the number of successful path searches in the first 100 rounds first increases slowly and then increases rapidly. This shows that the large ship using the method of the present application has learned to avoid collisions with static obstacles after 300 trainings. Among them, the optimal navigation path obtained by the method of the present application was learned for the 968th time. The reason for not continuing to optimize this path is that according to the greedy strategy, there is a certain probability that the ship will search for other paths. However, from the 969th time to the end of the training stage, the searched path was no better than the result of the 968th time. It should be noted that the number of times M that the ship successfully reached the target point changed from 22 times to 703 times, an increase of more than 30 times. This means that the method of the present application makes it more likely that the learned strategy converges to the optimal strategy. In summary, the path optimization method of the present application combines DQN with the physical model in the artificial potential field, which can enhance the learning efficiency of the algorithm.

[0140] Therefore, the path planned by the path optimization method adopted in steps S101-S105 of the present application not only improves the learning efficiency, but also improves the safety, economy and feasibility of the planned path. However, the planned path has 6 right-angle bends, that is, a track with a sudden change of 90° in heading, which is inconsistent with the actual navigation track of large ships.

[0141] Therefore, the smoothing trajectory is performed through steps S106-S107, as follows:

[0142] This application conducts an experiment on a large ship, and processes the above-planned path to meet the turning performance of the large ship. The main ship type parameters of the large ship are shown in Table 2. The parameters of the nonlinear Nomoto model are obtained as follows: K =0.21, T =107.78, =13.17, =16323.89. After studying the servo characteristics of the experimental object, the maximum rudder angle of the servo during the experiment was set to 35°, the maximum steering speed is set to 5° / s.

[0143]

[0144] A ship steady turning simulation test was carried out on the large ship. Figure 7As shown in the figure, it can be seen that the longitudinal tactical diameter in this simulation experiment is about 694m, which is 3.7 times the length of the ship, and the transverse tactical diameter is about 717m, which is 3.8 times the length of the ship, which is similar to the actual results of the large ship's steady turning experiment. The path planning data planned by the path optimization method of steps S101-S105 used in the above experiment is taken as an example for processing. Figure 8 As shown in the figure, the large ship of the experimental object started from the starting point, crossed various obstacles and finally sailed to the end point. It went through a total of 6 turns that were more in line with its maneuverability characteristics, and always maintained a safe distance of at least 150m from the obstacles.

[0145] In summary, the present application provides a large ship path planning method that integrates DQN and artificial potential fields. This method is divided into two parts: path optimization and feasibility enhancement. The navigation path planned by the path optimization part in the simulation experiment solves the local minimum and target unreachable problems of the traditional artificial potential field method, and the path search is more purposeful and the search efficiency is improved. In the feasibility enhancement stage, based on the mathematical model of the ship, the simulated navigation trajectory of the ship considering the response of the steering gear is obtained. This trajectory is then used for processing to obtain the final smooth path. After the optimal path is searched for feasibility enhancement, the path turns are smoother and meet the maneuvering characteristics of large ships. The path planned by this application not only has the three characteristics of safety, economy, and feasibility, but also improves the learning efficiency of the algorithm.

[0146] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A large ship path planning method integrating DQN and artificial potential field, characterized in that: The following steps are involved: Acquire navigation environment information of a navigation area, where the navigation area is the area between a starting point and a target point of the ship; Dividing the navigation area into a permitted navigation area and a restricted navigation area according to the navigation environment information, and obtaining grid map data of the navigation area; Constructing a resultant force field model based on an artificial potential field according to grid map data of the navigation area, wherein the resultant force field model includes a gravitational field sub-model and a repulsive field sub-model; Constructing a reward function based on the DQN algorithm according to the resultant force field model; Performing deep reinforcement learning search processing on the ship's navigation path according to the reward function to obtain first path planning data; Constructing a ship simulation model corresponding to the ship, performing a ship steady turn simulation on the ship simulation model, and obtaining ship simulation navigation track data; Performing smoothing processing on the first path planning data according to the simulated navigation trajectory data of the ship to obtain second path planning data, wherein the second path planning data is used to instruct the ship to navigate from the starting point to the target point; The expression of the gravitational field submodel is: ; in, is the gravitational coefficient, Indicates the Euclidean distance from the ship to the target point; The expression of the repulsive field submodel is: ; in, is the repulsion coefficient, is the Euclidean distance from the ship to the restricted navigation zone, The scope of influence of the restricted navigation zone; The expression of the reward function is: 。 2. According to claim 1, a large ship path planning method integrating DQN and artificial potential field is characterized in that: The deep reinforcement learning search process of the ship's navigation path is performed according to the reward function to obtain first path planning data, including: Constructing an initialized neural network model according to the reward function; Fitting the action value function according to the initialized neural network model to obtain environmental observation sequence data, and storing the environmental observation sequence data in an experience pool; The initialized neural network model is trained through the experience pool to obtain the trained main network model; Updating the fitted action-value function according to the main network model to obtain an updated action-value function; A search process is performed according to the updated action value function to obtain first path planning data.

3. A large ship path planning method integrating DQN and artificial potential field according to claim 2, characterized in that: The method of training the initialized neural network model through the experience pool to obtain the trained main network model also includes: A target network model based on the main network model is constructed, and the target network model copies the network parameters of the main network model according to the target time interval.

4. According to claim 2, a large ship path planning method integrating DQN and artificial potential field is characterized in that: In the step of performing search processing based on the updated action-value function to obtain the first path planning data, the updated action-value function is searched based on a time-varying greedy strategy to obtain the first path planning data.

5. According to claim 1, a large ship path planning method integrating DQN and artificial potential field is characterized in that: The constructing of a ship simulation model corresponding to the ship, performing a ship steady turn simulation on the ship simulation model, and obtaining ship simulation navigation track data includes: Construct a ship simulation model of the corresponding ship; obtaining a steering gear servo system associated with the turning performance of the vessel; According to the ship simulation model and the steering gear servo system, the ship simulation model is simulated for steady turning to obtain the position information of the ship when turning. The position information of the ship when turning is expanded according to the symmetry algorithm to obtain the ship simulation navigation track data.

6. A large ship path planning method integrating DQN and artificial potential field according to claim 1 or 5, characterized in that: The ship simulation model is a nonlinear Nomoto model.

7. A large ship path planning method integrating DQN and artificial potential field according to claim 5, characterized in that: The step of performing expansion processing on the position information of the ship when turning according to the symmetry algorithm includes: Obtaining course change trajectory information according to the position information of the ship when turning, wherein the course change trajectory information is trajectory information in which the course of the ship changes by 90° during navigation; Performing expansion processing on the course change trajectory information according to a symmetry algorithm to obtain expanded course change trajectory information; The expanded course change trajectory information is translated to the right-angle turn to obtain the ship's simulated navigation trajectory data.

Citation Information

Patent Citations

  • Two-time path planning method considering movement performance of unmanned surface vehicle

    CN108445894A

  • Ship path planning method and device based on clustering processing and artificial potential field

    CN112650237A