A ship collision avoidance decision-making method based on two-stage security reinforcement learning

Abstract

The application discloses a ship collision avoidance decision-making method based on two-stage safety reinforcement learning, which comprises the following steps: constructing a first-stage reinforcement learning ship collision avoidance decision-making model, which is used for a ship to take a collision avoidance action when the ship has other ships or obstacles within a first threshold value around the ship; constructing a second-stage safety reinforcement learning ship collision avoidance decision-making model, which is used for limiting a ship action space to ensure the safety of ship navigation when the ship has other ships or obstacles within a second threshold value around the ship; when the first threshold value condition is met, the trained first-stage reinforcement learning ship collision avoidance decision-making model is used for making a collision avoidance decision in a ship running process; and when the second threshold value condition is met, the trained second-stage safety reinforcement learning ship collision avoidance decision-making model is used for making a collision avoidance decision in the ship running process. The application improves the safety and economy of ship navigation and provides a safety guarantee for a reinforcement learning collision avoidance algorithm.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of ship navigation, and relates to a ship collision avoidance decision-making method based on two-stage safety reinforcement learning. BACKGROUND

[0002] With the vigorous development of global maritime traffic, the number of ships is also increasing rapidly. This growth trend makes maritime navigation face increasing challenges and risks. The harm caused by maritime accidents to personnel safety and property is also increasingly prominent. Therefore, through intelligent technology to realize the autonomous collision avoidance decision-making of ships, so as to reduce the influence of human factors on ship collision avoidance accidents, has become one of the hotspots widely concerned in the industry.

[0003] In the existing ship collision avoidance decision-making algorithm, the collision avoidance problem in high-density water area and complex water area is recognized as a difficulty in ship collision avoidance problem. There are many ships in high-density water area, and the navigation path is complex and interlaced, and the collision avoidance requirement is very high, which makes the traditional collision avoidance algorithm often perform poorly in such environment. The ordinary collision avoidance algorithm usually faces two main problems: one is that the modeling and prediction of the surrounding environment are insufficient, resulting in insufficient comprehensive collision avoidance strategy; the second is that in order to ensure safety, too conservative strategy is adopted, which excessively limits the freedom of action of the ship, significantly reduces the solution set, and makes the collision avoidance problem in the current scene more difficult. Rule-oriented collision avoidance algorithms, such as traditional methods based on collision avoidance heading angle, may have difficulty in responding flexibly when dealing with high-density ship flow, and even lead to the dilemma that the ship cannot effectively avoid collision.

[0004] In contrast, reinforcement learning algorithms exhibit unique advantages in dealing with collision avoidance problems in high-density and complex water areas. Reinforcement learning can adaptively learn the optimal collision avoidance strategy through a large number of interactions with the environment, and in the face of complex and variable navigation environment, it can solve the challenges that traditional algorithms cannot cope with in a more flexible way. In addition, the reinforcement learning algorithm does not rely on pre-designed rules, but gradually optimizes the collision avoidance behavior in a dynamic environment through a trial-and-error mechanism, which makes it perform better in highly complex navigation scenarios.

[0005] However, the main defect of the reinforcement learning collision avoidance algorithm is that it is essentially a "black box" model, lacking in explainability. The learning process of the algorithm is complex and it is difficult to clearly explain the specific decision basis, which may reduce its reliability in actual application. Especially in the key safety field of ship collision avoidance, the transparency and verifiability of the decision-making process are crucial, and the unexplainability of the reinforcement learning algorithm makes the decision-making process difficult to be trusted by users and regulatory authorities. This is also one of the biggest challenges faced by reinforcement learning in the practical application of production environment.

[0006] The existing ship collision avoidance algorithm has made great progress in recent years. However, despite the continuous progress of technology, there are still several limitations that make the existing ship collision avoidance algorithm face challenges in practical applications. The current mainstream collision avoidance algorithm can be divided into path planning-based collision avoidance algorithm and machine learning-based collision avoidance algorithm.

[0007] For path planning-based collision avoidance algorithms, such as A* algorithm, velocity obstacle method, artificial potential field method, and heuristic algorithm, etc. The advantage of such algorithms is to calculate the optimal collision avoidance path through mathematical model, which has explainability. These optimization algorithms can find the optimal path under certain conditions, but there are still some problems. First, the calculation complexity is high, and in the case of multiple ship collision avoidance, the complexity of calculating the optimal path increases exponentially, which makes the algorithm difficult to meet the real-time requirements in practical applications. In actual navigation, ships need to make quick decisions, and the high calculation demand of optimization algorithms may not be able to provide effective path planning in time. Second, these algorithms often ignore the navigation economy, and most optimization algorithms mainly focus on avoiding collision, but ignore the navigation economy such as fuel consumption and time cost. Lack of comprehensive consideration of economy may lead to the algorithm unable to balance safety and efficiency in practice, reducing the value of actual application.

[0008] For artificial intelligence-based collision avoidance algorithms, especially reinforcement learning collision avoidance algorithms, their application in ship collision avoidance field has gradually increased in recent years. Through reinforcement learning, the system can learn collision avoidance strategies through a large amount of training data. However, existing reinforcement learning collision avoidance algorithms are all single reinforcement learning in the whole process, and the agent often cannot balance navigation economy and navigation safety. In addition, due to the special black box characteristics of reinforcement learning and the serious consequences of ship collision, reinforcement learning collision avoidance algorithm is difficult to be put into actual production activities.

[0009] By comparing and analyzing the existing two types of collision avoidance algorithms from the aspects of calculation complexity, real-time response, explainability, safety guarantee, etc., it can be found that the existing collision avoidance algorithms largely ignore the navigation economy, making the autonomous surface ships carrying the algorithm always avoid the object ship too conservatively. Secondly, many algorithms assume that ships can immediately change course or speed, without fully considering the physical characteristics of ships such as inertia, turning curvature, etc., and the application of ship motion model is limited.

[0010] From the above, it can be found that if the collision avoidance algorithm needs to be put into actual production application, the following problems need to be solved:

[0011] ① The collision avoidance algorithm needs to have the ability to handle complex sea areas with multiple ship encounters;

[0012] ② For artificial intelligence-based collision avoidance algorithms, there needs to be a means of explanation or guarantee of safety;

[0013] ③ During the long period of time in the future when autonomous surface vessels and manually operated vessels will coexist at sea, the economic efficiency of navigation should be taken into account, and overly conservative actions should not be taken when encountering other vessels.

[0014] ④ When constructing a collision avoidance algorithm, it is necessary to take into account as many real-world navigation situations as possible, including factors such as ship motion models and collision detection. Summary of the Invention

[0015] To address the aforementioned problems, the technical solution adopted in this invention is: a ship collision avoidance decision-making method based on two-stage safety reinforcement learning, comprising the following steps:

[0016] Design a ship motion model based on a set of reinforcement learning scenarios and parameters;

[0017] Based on the ship motion model and combined with reinforcement learning algorithm, we designed the reward function for compliance with international maritime collision avoidance rules, the reward function for navigation economy, and the reward function for minimum encounter distance, and constructed the first-stage reinforcement learning ship collision avoidance decision model, which is used to make the ship take collision avoidance actions when there are other ships or obstacles in the surrounding area within the first threshold.

[0018] The first-stage reinforcement learning ship collision avoidance decision model is trained to obtain a well-trained first-stage reinforcement learning ship collision avoidance decision model.

[0019] Based on the ship motion model and the action probability distribution of the first-stage reinforcement learning ship collision avoidance decision model, the weight of each level 3 action is determined and the proportion of dangerous level 3 actions is used to design the reward function of the second stage and construct the second-stage safety reinforcement learning ship collision avoidance decision model. This model is used to limit the ship's action space to ensure the safety of ship navigation when there are other ships or obstacles in the surrounding area within the second threshold.

[0020] The second-stage safety reinforcement learning ship collision avoidance decision model is trained to obtain a well-trained second-stage safety reinforcement learning ship collision avoidance decision model.

[0021] When a ship encounters another ship or obstacle under the first threshold condition, the pre-trained first-stage reinforcement learning ship collision avoidance decision model is used to make collision avoidance decisions during the ship's navigation process.

[0022] When a ship encounters other ships or obstacles under the second threshold condition, a pre-trained second-stage safety reinforcement learning ship collision avoidance decision-making model is used to make collision avoidance decisions during the ship's navigation process.

[0023] Furthermore: the first threshold condition refers to the maximum detection radius of the radar carried by the ship;

[0024] The second threshold condition refers to the collision risk value between the subject ship and other ships or obstacles.

[0025] Further, the international maritime collision avoidance rule compliance reward function is used to represent that 200 positive rewards are given to the subject ship if it complies with the rules, or -200 negative rewards are given if it does not comply with the rules, when multiple ships meet, if there is a ship that requires the subject ship to drive the avoidance responsibility, the negative reward of the subject ship as a straight-ahead ship is cancelled.

[0026] The expression of the international maritime collision avoidance rule compliance reward function is as follows:

[0027]

[0028] In the formula, COLREG reward represents the international maritime collision avoidance rule compliance reward function; C success represents the number of ships that avoid according to the international maritime collision avoidance rules; C stand-on represents the number of ships that require the subject ship to avoid in the two-ship meeting situation; A own_ship represents the action set of the subject ship, the target heading is always 1 in this reward, otherwise it is -1.

[0029] Further, the sailing economy reward function is used to define the angle difference between the sailing direction of the subject ship and the direction of the terminal point.

[0030] The expression of the sailing economy reward function is as follows:

[0031]

[0032] In the formula, economy reward represents the sailing economy reward function; C dest is the direction of the terminal point relative to the subject ship; is the true heading of the subject ship; and α is the economy coefficient of the intelligent agent.

[0033] Further, the minimum encounter distance reward function is used to calculate the reward given by the minimum encounter distance between ships.

[0034] The expression of the minimum encounter distance reward function is as follows:

[0035]

[0036]

[0037] In the formula, DCPA reward represents the minimum encounter distance reward function, C other_ship is the number of other ships in the environment.

[0038] Further, the first-stage reinforcement learning ship collision avoidance decision model comprises:

[0039] A first-stage scene generation module for generating a scene in which a ship may encounter;

[0040] A first-stage state identification module for classifying a ship in an initial environment, identifying a ship encounter scene based on the relative bearing, distance of the ship from the subject ship, and the heading of the ship;

[0041] A first-stage reward function module for outputting a reward value according to a plurality of reward items, i.e., an international maritime collision avoidance rule compliance reward function, a navigation economy reward function, and a minimum encounter distance reward function;

[0042] A first-stage reward statistics module for performing statistics on the reward value output by the first-stage reward function module;

[0043] A first-stage reinforcement learning algorithm module for outputting a decision on whether to update the current environment based on the reward value output by the first-stage reward function module, the scene in which a ship may encounter output by the scene generation module, and the current scene based on a reinforcement learning algorithm;

[0044] A first-stage environment update module for updating the environment during the operation of a ship based on the decision output by the reinforcement learning algorithm module, the ship encounter scene output by the state identification module, and a ship motion model.

[0045] Further, the second-stage safety reinforcement learning ship collision avoidance decision model comprises:

[0046] A second-stage scene generation module for generating a scene in which a ship may encounter;

[0047] A second-stage state identification module for classifying a ship in an initial environment, identifying a ship encounter scene based on the relative bearing, distance of the ship from the subject ship, and the heading of the ship;

[0048] A collision detection module for calculating the collision possibility within a future time step;

[0049] A three-level tree decision module for predicting the next three steps of actions based on the first-stage reinforcement learning ship collision avoidance decision model, calculating the three-level nodes with collision risks using the collision detection module and marking them; if a node is determined to be dangerous, all of its child nodes are marked as having collision risks, the probability distribution of actions of the first-stage reinforcement learning model is used to derive the probability of each node, and the three-level nodes are assigned the proportion; and dangerous actions are deleted, thereby playing a safety guarantee role;

[0050] The reward function module of the second-stage safety reinforcement learning: used to give a reward to the primary action based on the safety of the three-level node and the probability of each action output by the three-level tree decision module;

[0051] The second-stage reward statistics module: used to count the reward output by the reward function module of the second-stage safety reinforcement learning;

[0052] The second-stage reinforcement learning algorithm module: used to output a decision of whether to update the current space based on the reward value output by the reward function module of the second-stage safety reinforcement learning, the scenario that the ship may encounter output by the scenario generation module, and the action output by the three-level tree decision module based on the reinforcement learning algorithm;

[0053] The update environment module: used to update the current ship sailing environment based on the ship motion model, the ship encounter scenario output by the second-stage state recognition, and the decision of whether to update the current space output by the second-stage reinforcement learning algorithm module.

[0054] Further, the reward function expression of the second-stage safety reinforcement learning is as follows:

[0055]

[0056] In the formula, Reward represents the reward function value corresponding to the primary action; β represents whether the three-level action has a collision risk, and is 1 if it has a collision risk, otherwise is 0; A event is “the secondary action i occurs in this state”; B event is “the three-level action j occurs in this state”.

[0057] Further, the process of calculating the three-level nodes with collision risks by the collision detection module based on the three-level nodes in the three-level tree decision module is as follows:

[0058] A three-level decision tree model is constructed for the current scenario and is traversed and calculated; the three-level nodes with collision risks are calculated and marked by the collision detection module based on the three-level nodes

[0059] If a decision node is determined to have a collision, all its child nodes are regarded as collision nodes,

[0060] If all the child nodes of a node are collision nodes, the node itself is also determined to have a collision, and the collision node is defined as a dangerous action;

[0061] The dangerous action is deleted, the ship action space is updated, and the safety guarantee effect is achieved.

[0062] Further, when the number of collision nodes in the three-level decision tree is greater than or equal to the threshold number of dangerous nodes, the first-stage reinforcement learning ship collision avoidance decision model trained is adopted, and a collision avoidance action is taken.

[0063] When the number of collision nodes in the three-level decision tree is less than the threshold number of dangerous nodes, the trained second-stage model is used to make ship collision avoidance decisions, and the ship action space is limited to ensure the safety of ship navigation.

[0064] The ship collision avoidance decision-making method based on two-stage safety reinforcement learning has the following advantages:

[0065] Based on the two-stage reinforcement learning method, a number of modules are added to assist decision-making, and the ship decision-making space is dynamically limited to ensure its safety, and a method for building a ship collision avoidance algorithm is proposed;

[0066] The method models the collision avoidance process of the ship as a Markov decision process, and designs different reward functions and action spaces for each stage, so that the agent can obtain effective collision avoidance decisions from the environment through the neural network (reinforcement learning algorithm). In addition, the method ensures the safety of the decision through multi-level decision trees and action space restrictions.

[0067] The first stage of the method improves the navigation economy under non-high-risk conditions through reinforcement learning, and the second stage significantly improves the collision avoidance performance in complex water areas or multi-ship encounter situations through safe decision-making induction and action space restriction. This provides a practical method for applying reinforcement learning collision avoidance algorithms to engineering practice.

[0068] The two-stage safety reinforcement learning collision avoidance method designed by the application achieves the following results: first, the method can ensure that the ship realizes efficient navigation that meets economic benefits under normal conditions; second, in high-risk scenarios, the algorithm builds multi-level decision trees and action space constraints through a rigorous three-level tree decision module and collision detection module to exclude potential dangerous paths and select safe and robust collision avoidance decisions for the ship. This innovative dual reinforcement learning mechanism not only retains the flexibility of reinforcement learning algorithms to adapt to scenarios, but also provides safety and reliability support for the application of such algorithms in ship collision avoidance, providing a new solution to the ship collision avoidance problem in high-density complex water scenarios.

[0069] The application provides a ship autonomous collision avoidance decision-making method based on reinforcement learning, which has the following beneficial effects.

[0070] First, autonomous learning and dynamic adaptation: the method can dynamically adjust the collision avoidance strategy through continuous interaction with the complex dynamic environment, avoiding the limitations of relying on pre-designed complex rules or models, and is particularly suitable for navigation scenarios in high-density water areas and complex water areas. Through the accumulation of experience through reinforcement learning, the agent can make optimal choices when faced with complex nonlinear decision-making problems, effectively solving the instability of traditional algorithms in complex environments.

[0071] Secondly, the compatibility of two-stage reinforcement learning: the application realizes the simultaneous consideration of economy and safety in different navigation stages through two-stage reinforcement learning. The first stage focuses on optimizing navigation efficiency based on compliance with international maritime collision regulations, enabling the ship to flexibly perform its duties as a yielding ship. The second stage focuses on ensuring safety in high-risk scenarios, especially when the other ship fails to fulfill its avoidance obligations. This dual strategy design ensures the ship's collision avoidance capability in complex waters and improves the compatibility of the method, allowing automatic switching to the appropriate decision-making mode according to different scenarios.

[0072] Finally, the robustness of the second-stage safety reinforcement learning: this stage of reinforcement learning not only ensures the safety of ship decision-making through safe decision induction and action space restriction, but also performs well in complex water areas or high-risk scenarios such as entering and exiting ports. By introducing more navigation information such as ship parameters, motion capabilities, water depth topography, and other ship intelligent algorithms, the robustness and practical applicability of collision avoidance decisions are further enhanced, providing solid technical support for the practical engineering application of reinforcement learning collision avoidance algorithms.

[0073] In summary, the application has certain beneficial effects in the field of ship intelligent navigation technology. It improves the safety and economy of ship navigation, provides a safety guarantee for reinforcement learning collision avoidance algorithms, and provides a research approach for the safety and reliability of ship intelligent navigation systems. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0075] Figure 1 is a flowchart of a ship collision avoidance decision-making method based on two-stage safety reinforcement learning;

[0076] Figure 2 is a design flowchart of the first-stage reinforcement learning ship collision avoidance decision-making model;

[0077] Figure 3 is a design flowchart of the second-stage safety reinforcement learning ship collision avoidance decision-making model;

[0078] Figure 4 is a quantitative diagram of the international maritime collision regulations for meeting scenarios;

[0079] Figure 5A schematic diagram of the ship size;

[0080] Figure 6 A decision tree model;

[0081] Figure 7 Model performance in a single-ship encounter scenario, where (a) is a single-ship encounter situation, (b) is a single-ship small-angle right crossing situation, (c) is a single-ship medium-angle right crossing situation, and (d) is a single-ship large-angle right crossing situation;

[0082] Figure 8 Model performance in a multi-ship encounter scenario, where (a) is a four-ship encounter situation, and (b) is a multi-ship encounter situation. DETAILED DESCRIPTION

[0083] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict, and the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0084] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The description of the at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0085] Figure 1 A flowchart of a ship collision avoidance decision-making method based on two-stage safety reinforcement learning;

[0086] A ship collision avoidance decision-making method based on two-stage safety reinforcement learning, which is divided into two stages. The reinforcement learning in the first stage focuses on the sailing economy of the ship and the compliance with rules, aiming to make the ship efficiently complete the sailing task while complying with the rules. The safety reinforcement learning in the second stage focuses on the safety of sailing, and further ensures the safety of its decision-making by limiting the action space. Therefore, it is necessary to set a mode conversion condition related to safety to determine when to use which decision-making method.

[0087] The method comprises the following steps:

[0088] S1: design a ship motion model based on a reinforcement learning scenario set and parameters;

[0089] S2: Based on the ship motion model, combined with the reinforcement learning algorithm, the international maritime collision avoidance rule compliance reward function, the sailing economy reward function, and the minimum encounter distance reward function are designed, and a first-stage reinforcement learning ship collision avoidance decision model is constructed, which is used for the ship to take collision avoidance actions when there are other ships or obstacles within the first threshold value around the ship; In this process, the economy of sailing must be considered as much as possible to avoid overly conservative collision avoidance decisions; and the weights of each action for the second-stage safe reinforcement learning are allocated;

[0090] S3: The first-stage reinforcement learning ship collision avoidance decision model is trained to obtain a trained first-stage reinforcement learning ship collision avoidance decision model;

[0091] S4: Based on the ship motion model and the action probability distribution of the first-stage reinforcement learning ship collision avoidance decision model, the proportions of each three-level action are determined, and the proportions of dangerous three-level actions are designed, and a second-stage safe reinforcement learning ship collision avoidance decision model is constructed, which is used for limiting the ship action space to ensure the safety of ship sailing when there are other ships or obstacles within the second threshold value around the ship;

[0092] S5: The second-stage safe reinforcement learning ship collision avoidance decision model is trained to obtain a trained second-stage safe reinforcement learning ship collision avoidance decision model;

[0093] S6: When the ship and other ships or obstacles are in the first threshold condition, the trained first-stage reinforcement learning ship collision avoidance decision model is used for collision avoidance decision during the ship sailing process.

[0094] When the ship and other ships or obstacles are in the second threshold condition, the trained second-stage safe reinforcement learning ship collision avoidance decision model is used for collision avoidance decision during the ship sailing process.

[0095] The steps S1 / S2 / S3 / S4 / S5 / S6 are sequentially executed;

[0096] Further, the first threshold condition refers to the maximum detection radius of the radar carried by the ship; The maximum detection radius in this embodiment is 6 nautical miles;

[0097] The second threshold condition refers to the determination of the collision risk value between the ship and other ships or obstacles.

[0098] Figure 2 is a design flowchart of the first-stage reinforcement learning ship collision avoidance decision model;

[0099] The first-stage reinforcement learning ship collision avoidance decision model comprises:

[0100] The first-stage scene generation module is used for generating possible scenes encountered by the ship;

[0101] The first stage state recognition module: classifying the other ships in the initial environment, identifying the ship encounter scenarios based on the relative bearing, distance of the other ships to the own ship and the heading of the other ships;

[0102] The first stage reward function module: used for outputting reward values according to multiple reward items, i.e., an international maritime collision avoidance rule compliance reward function, a navigation economy reward function and a minimum encounter distance reward function;

[0103] The first stage reward statistics module: used for performing statistics on the reward values output by the first stage reward function module;

[0104] The first stage reinforcement learning algorithm module: used for outputting a decision on whether to update the current environment based on the reward values output by the first stage reward function module, the scenarios in which the ships may encounter output by the scenario generation module and the current scenario based on a reinforcement learning algorithm;

[0105] The first stage environment update module: used for updating the environment in the running process of the ships based on the decision output by the reinforcement learning algorithm module, the ship encounter scenarios output by the state recognition module and the motion model of the ships.

[0106] The process of training the first stage reinforcement learning ship collision avoidance decision-making model;

[0107] In the training process of reinforcement learning, the scenario generation module, the state recognition module, the system motion model and the reward function are integrated into a unified environment, and the reinforcement learning algorithm is accessed for training.

[0108] Firstly, the environment is constructed and initialized, the above modules are integrated, and a complete training platform is formed. The environment can generate diversified training scenarios in real time and feedback the state information of the system. At the beginning of each training, the environment has a certain random initialization capability, creates new scenarios and states, and provides a variable and challenging training background for the agent.

[0109] Next, the agent is trained. At each time step, the agent selects an action according to the selected reinforcement learning algorithm based on the current state. Whether it is a policy-based algorithm (such as PPO, TRPO) or a value-based algorithm (such as DQN, DDPG), the agent will generate an action using the current policy or value function, which enables the agent to explore different decision paths during the training process. Subsequently, the selected action is applied to the environment, causing the system state to change, and the environment updates the state according to the execution result and calculates the reward value, reflecting the performance of the agent in the state.

[0110] During training, the agent collects experiences of states, actions, and rewards to optimize its decision-making strategy. For policy optimization algorithms like PPO or A3C, these experiences are used to update the policy. For value function optimization algorithms like DQN, these experiences are used to update the Q-value function. Reinforcement learning algorithms ensure that the agent gradually improves its strategy through different updating methods, ultimately learning to make optimal decisions in different scenarios.

[0111] Finally, during training, the reward statistics module will regularly monitor the cumulative reward of the agent. As training progresses, the cumulative reward of the agent should gradually increase, indicating that its strategy is being optimized. By calculating the change in the policy or value function, we can determine whether the training has converged. If the policy or value function changes very little after multiple updates, it indicates that the agent has found a relatively stable decision-making strategy, and the training can be considered to have achieved the desired effect.

[0112] In the first phase of reinforcement learning, three reward functions are designed:

[0113] Regarding the international maritime collision avoidance rule compliance reward (event reward):

[0114] In the international maritime collision avoidance rules, Article 13 requires that when two ships are in a passing situation, the overtaking ship should give way to the overtaken ship; Article 14 requires that when two ships are in a meeting situation, they should each turn to the right, thus passing from the left side of the other ship; Article 15 requires that when two ships are in a crossing situation, the ship with the other ship on the right side should give way to the other ship, and if the environment permits, it should also avoid crossing in front of the other ship; in addition, if the ship is a straight-ahead ship, it should also fulfill the obligation of maintaining speed and direction.

[0115] According to the description of the international maritime collision avoidance rules, the international maritime collision avoidance rule compliance reward function is used to represent that if the ship follows the rules, it gives a positive reward of 200, otherwise it gives a negative reward of -200, and when multiple ships meet, if the other ship requires the ship to perform the avoidance responsibility, the negative reward for being a straight-ahead ship is cancelled.

[0116] The expression of the international maritime collision avoidance rule compliance reward function is as follows:

[0117]

[0118] In the formula: COLREG reward represents the international maritime collision avoidance rule compliance reward function; C success represents the number of ships that follow the international maritime collision avoidance rules for avoidance; C stand-on represents the number of ships that require the ship to perform avoidance in a meeting situation; A own_shipThe action set of the ship, the target heading is always the end point direction in this reward, otherwise -1.

[0119] The reward function of sailing economy (step by step reward) ;

[0120] The sailing economy reward function is used to define the angle difference between the ship's heading and the end point; the proportion of this reward directly determines the agent's emphasis on sailing economy, the greater the proportion, the smaller the circumnavigation distance, and vice versa, the smaller the proportion, the more emphasis on sailing safety.

[0121] The expression of the sailing economy reward function is as follows:

[0122]

[0123] In the formula: economy reward The sailing economy reward function; C dest The direction of the end point relative to the ship; The true heading of the ship; alpha is the economy coefficient of the agent; here alpha = 0.0035.

[0124] The reward function of the minimum encounter distance;

[0125] The minimum encounter distance reward function is used to calculate the reward given by the minimum encounter distance between ships; in order to avoid the situation of two ships being too close, attention should be paid to not making this reward too large.

[0126] The expression of the minimum encounter distance reward function is as follows:

[0127]

[0128]

[0129] In the formula: DCPA reward The minimum encounter distance reward function, C other_ship The number of other ships in the environment.

[0130] Figure 3 The design flowchart of the second stage safety reinforcement learning ship collision avoidance decision model;

[0131] The second stage safety reinforcement learning ship collision avoidance decision model comprises:

[0132] The second stage scene generation module: used for generating the scene that the ship may encounter;

[0133] The second stage state recognition module: used for classifying other ships in the initial environment, based on the relative bearing, distance of the ship and the heading of the other ship, and identifying the ship encounter scene;

[0134] collision detection module: for calculating the possibility of collision in the future time step;

[0135] third-level tree decision module: for predicting the action of the next three steps based on the first-stage reinforcement learning ship collision avoidance decision model, taking the third-level node as the statistical basis, calculating the third-level node with collision risk by using the collision detection module and marking; if a node is determined to be dangerous, all its child nodes are marked as having collision risk, and the probability distribution of the first-stage reinforcement model action is used to derive the probability of each node, and the third-level node is given the proportion of the proportion; and delete the dangerous action, play a safety guarantee role;

[0136] reward function module of the second-stage safety reinforcement learning: for giving rewards to the first-stage action based on the safety of the third-level node of each action output by the third-level tree decision module and the probability of the third-level node;

[0137] second-stage reward statistics module: for counting the rewards output by the reward function module of the second-stage safety reinforcement learning;

[0138] second-stage reinforcement learning algorithm module: for outputting the decision of whether to update the current space based on the reward value output by the reward function module of the second-stage safety reinforcement learning, the ship encounter scenario output by the scene generation module, and the action output by the third-level tree decision module based on the reinforcement learning algorithm;

[0139] update environment module: for updating the current ship driving environment based on the ship motion model, the ship encounter scenario output by the second-stage state recognition, and the decision of whether to update the current space output by the second-stage reinforcement learning algorithm module.

[0140] The second-stage reward function is designed as a safety reward function based on a decision tree:

[0141] The expression of the second-stage reward function is as follows:

[0142]

[0143] In the formula, Reward represents the reward function value corresponding to the first-stage action; β represents whether the third-level action has collision risk, 1 for collision risk and 0 otherwise; A event is “the occurrence of second-level action i in this state”; B event is “the occurrence of third-level action j in this state”.

[0144] The process of the third-level tree decision module taking the third-level node as the statistical basis and calculating the third-level node with collision risk by using the collision detection module is as follows:

[0145] A three-level decision tree model is constructed for the current scene and traversed to calculate; the three-level nodes are calculated for collision risk using the collision detection module and marked

[0146] If a decision node is determined to have a collision, all its child nodes are considered collision nodes,

[0147] If all child nodes of a node are collision nodes, the node itself is also determined to have a collision, and the collision node is defined as a dangerous action;

[0148] Delete the dangerous action to update the ship's action space and ensure safety;

[0149] When the number of collision nodes in the three-level decision tree is greater than or equal to the threshold number of dangerous nodes, the trained first-stage reinforcement learning ship collision avoidance decision model is used to take collision avoidance actions,

[0150] When the number of collision nodes in the three-level decision tree is less than the threshold number of dangerous nodes, the trained second-stage reinforcement learning ship collision avoidance decision model is used to limit the ship's action space to ensure the safety of ship navigation.

[0151] The process of training the second-stage safety reinforcement learning ship collision avoidance decision model is as follows:

[0152] The scene generation module, state recognition module, ship motion model, three-level decision tree, and second-stage reward function are integrated into a second-stage safety reinforcement learning environment, and the environment is connected to the PPO algorithm for safety reinforcement learning training.

[0153] During the reinforcement learning training process, the scene generation module, state recognition module, ship motion model, three-level decision tree module, and second-stage safety reinforcement learning reward function are integrated into a unified environment, and the reinforcement learning algorithm is connected for training.

[0154] First, the environment is constructed and initialized, the above modules are integrated, and a complete training platform is formed. The environment can generate diverse training scenes in real time and feedback the state information of the system. At the beginning of each training, the environment has certain random initialization capabilities, creating new scenes and states to provide a variable and challenging training background for the agent.

[0155] Next, the training of the agent is performed. At each time step, the agent selects an action based on the selected reinforcement learning algorithm through the current state. Whether it is a policy-based algorithm (such as PPO, TRPO) or a value-based algorithm (such as DQN, DDPG), the agent will generate an action using the current policy or value function, which allows the agent to explore different decision paths during training. Subsequently, the selected action is applied to the environment, causing the system state to change, and the environment updates the state and calculates the reward value reflecting the performance of the agent in that state.

[0156] During training, the agent continuously optimizes its decision-making strategy by collecting experiences of states, actions, and rewards. For policy optimization algorithms such as PPO or A3C, these experiences are used to update the policy. For value function optimization algorithms such as DQN, these experiences are used to update the Q-value function. Reinforcement learning algorithms use different update methods to ensure that the agent gradually improves its strategy and eventually learns to make optimal decisions in different scenarios.

[0157] In addition, through the prediction process of the three-level decision tree for the next three time steps based on the current time, the feasible range of the action space at the current time is returned, limiting the action space within a safe range.

[0158] Finally, during training, the reward statistics module will periodically monitor the cumulative reward of the agent. As training progresses, the cumulative reward of the agent should gradually increase, indicating that its strategy is being optimized. By calculating the change in the policy or value function, we can determine whether the training has converged. If the policy or value function changes very little after multiple updates, it indicates that the agent has found a relatively stable decision-making strategy, and at this point, we can determine that the training has achieved the desired effect.

[0159] Through the above two-stage reinforcement learning training process, the agent ultimately realizes practical application in ship collision avoidance. When none of the reinforcement learning invocation conditions are met, the ship sails normally along the established route; if the first-stage condition is met but the second-stage condition is not met, the first-stage decision is invoked; if the opposite is true, the second-stage decision is invoked. When both conditions are met, the second-stage safe decision is prioritized.

[0160] Embodiment 1:

[0161] A ship collision avoidance decision-making method based on two-stage safe reinforcement learning includes the following steps:

[0162] S1: Design a ship motion model based on a reinforcement learning scenario set and parameters;

[0163] For the first stage, define random 1-5 ships as the test ships, which are randomly distributed in the angle relative to the host ship, the distance is in the interval of 4-6 nautical miles, the heading is in the interval of ±20 degrees facing the host ship's heading, and it is assumed that the other ship is a non-acting ship with speed and direction preservation.

[0164] For the second stage, define random 1-5 ships as the test ships, which are randomly distributed in the angle relative to the host ship, the distance is in the interval of 1-2 nautical miles, the heading is in the interval of ±10 degrees facing the host ship's heading, and it is assumed that the other ship is a non-acting ship with speed and direction preservation.

[0165] The speed of the host ship is defined as 10 knots, and the size is as follows Figure 5 The action space of reinforcement learning is the expected heading at this time step. The state space is the position, distance, heading of all other ships relative to the host ship, and the type of encounter situation formed with the host ship in the environment. The time step for environment update and decision-making is 80s.

[0166] The ship motion model uses the Nomoto model based on the fourth-order Runge-Kutta numerical integration method to simulate the motion process of the ship, including the evolution of position, heading, speed and other states.

[0167] This ship motion model updates the state according to the following formula:

[0168]

[0169]

[0170]

[0171] In the formula: x is the x coordinate of the ship in the plane; y is the y coordinate of the ship in the plane; is the heading angle of the ship (the angle with the x axis); u is the longitudinal speed of the ship; v is the lateral speed of the ship; r is the angular speed of the ship.

[0172] According to the dynamics equation and PID control equation of the Nomoto model. The lateral speed v and angular speed r of the ship are controlled by the following differential equation:

[0173]

[0174] δ∈[δ min , δ max ]

[0175] In the formula: a 11 , a 12 , a 21 , a 22 , b 11 , b 21is the motion parameter of the ship, which describes the influence of the damping and inertia effect of the ship on the lateral velocity and the steering angular velocity; is the target heading; K p is the proportional gain coefficient; K d is the differential gain coefficient; δ min and δ max represent the minimum and maximum values of the input rudder angle of the ship, respectively.

[0176] The model uses the fourth-order Runge-Kutta method to approximately solve the differential equation, which can provide high precision and ensure the stability and accuracy of the ship motion model. The update steps are as follows:

[0177] k1 = h·f(t n , y n )

[0178]

[0179] k4 = h·f(t n +h, y n +k3)

[0180]

[0181] wherein k1, k2, k3, k4 represent the estimated values of the four slopes in the Runge-Kutta method; h is the time step; y n is the current state of the ship, including position, heading angle, velocity, rudder angle, etc. variables; t n is the current time; f is the state update function of the system;

[0182] S2: Based on the ship motion model, combined with the reinforcement learning algorithm, the international maritime collision avoidance rule compliance reward function, the navigation economy reward function, and the minimum encounter distance reward function are designed, and the first stage reinforcement learning ship collision avoidance decision model is constructed. When there are other ships or obstacles within the first threshold around the ship, the ship takes collision avoidance action; In this process, the economy of navigation must be considered as much as possible to avoid too conservative collision avoidance decision; and weight distribution is performed for each action of the second stage safety reinforcement learning;

[0183] S3: The first stage reinforcement learning ship collision avoidance decision model is trained to obtain the trained first stage reinforcement learning ship collision avoidance decision model; In the first stage reinforcement learning training, if there is no other obstacle or ship within 6 nautical miles around the ship or after 200 time steps, the current training round is ended to obtain the action output probability distribution of the intelligent agent in the whole process.

[0184] S4: determining the proportion of each tertiary action based on the ship motion model and the action probability distribution of the first-stage reinforcement learning ship collision avoidance decision model, and designing a reward function for the second stage according to the proportion of dangerous tertiary actions, and constructing a second-stage safe reinforcement learning ship collision avoidance decision model for limiting the ship action space to ensure the safety of ship navigation when there are other ships or obstacles within the second threshold around the ship;

[0185] The tertiary decision tree structure used by the second-stage safe reinforcement learning ship collision avoidance decision model is as shown in Figure 6

[0186] For each step of action, a decision tree of three depths is established, and there are 72 sub-nodes under each node according to the action space, and there are three layers in total,

[0187] For the secondary nodes and tertiary nodes, the specific scene and state space should be updated based on the previous action as the premise, and the action probability is obtained from the first-stage reinforcement learning model.

[0188] The construction process of the reward function of the second-stage safe reinforcement learning is as follows:

[0189] First, a tertiary decision tree model is constructed for the current scene to predict the next three steps of action, and the collision detection module is used to calculate and mark the tertiary nodes with collision risk. In this process, if a node is determined to be dangerous, all its sub-nodes are marked as having collision risk. Then, the first-stage reinforcement model action probability distribution is used to obtain the probability of each node, and the proportion of the tertiary nodes is given. Finally, the safety of the tertiary nodes of each action and the probability of each action are used as the basis to give a reward to the primary action;

[0190] Secondly, if a primary action is determined to have collision risk or the sum of the collision avoidance probabilities of the sub-nodes of the primary action is 1, the primary action is temporarily removed from the agent action space, so as to ensure the safety of the safe reinforcement learning ship collision avoidance decision algorithm.

[0191] In the second-stage safe reinforcement learning, if there are no other obstacles or ships within 2 nautical miles around the ship or the ship has an avoidance accident, it indicates that the current training round is over. This mechanism aims to ensure that when the ship is in a non-urgent state and does not need further decision, the current task can be effectively terminated, saving computing resources and improving training efficiency.

[0192] S5: training the second-stage safe reinforcement learning ship collision avoidance decision model to obtain a trained second-stage safe reinforcement learning ship collision avoidance decision model;

[0193] S6: integrating and testing the two-stage safe reinforcement learning model: ​

[0194] First, an initial scene is defined, the ship position and heading are random, and it should be noted that the number of other ships within the limited range of two-stage reinforcement learning should not exceed the observation space dimension.

[0195] When there are no other obstacles or ships within 6 nautical miles of the ship, the ship sails towards the target heading; when there are other ships or obstacles within 6 nautical miles, the ship starts the first stage of reinforcement learning and takes collision avoidance actions; when there are other ships or obstacles within 2 nautical miles, the two-stage reinforcement learning model is fully started, at this time if the number of dangerous nodes in the decision tree is less than 10%, the collision avoidance decision of the first stage of reinforcement learning is used, otherwise the second stage of safety reinforcement learning is used, at this time no matter which model is used, three-layer decision tree should be constructed and the action space should be limited to ensure the safety of navigation.

[0196] For single-ship collision avoidance scenarios, this algorithm can complete economic and rule-compliant collision avoidance strategies, such as Figure 7 shown, where (a) is a single-ship encounter situation, (b) is a single-ship small-angle right crossing situation, (c) is a single-ship medium-angle right crossing situation, and (d) is a single-ship large-angle right crossing situation.

[0197] The specific parameters of the scene are shown in Table 1.

[0198] Scenario His ship cross coordinate (m) His ship longitudinal coordinate (m) His ship heading (deg) Type of encounter (a) 0 11110 Due south Meeting (b) 2520 10819 South by West 14° Right crossing (c) 8290 7396 West by South 12° Right crossing (d) 10932 1972 West by North 40° Right crossing

[0199] When the number of surrounding ships increases, the model can exhibit safety properties under rule guidance, and can still maintain safe navigation in high-density and complex water scenarios. As shown in Figure 8 , where (a) is a four-ship encounter situation, and (b) is a multi-ship encounter situation.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacements for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A ship collision avoidance decision-making method based on two-stage security reinforcement learning, characterized in that: The method comprises the following steps: designing a ship motion model based on a reinforcement learning scene set and parameters; designing an international maritime collision avoidance rule compliance reward function, a navigation economy reward function, and a minimum encounter distance reward function based on the ship motion model and in combination with a reinforcement learning algorithm to construct a first-stage reinforcement learning ship collision avoidance decision model for a ship to take collision avoidance actions when there are other ships or obstacles within a first threshold around the ship; training the first-stage reinforcement learning ship collision avoidance decision model to obtain a trained first-stage reinforcement learning ship collision avoidance decision model; determining the proportions of various tertiary actions based on the action probability distribution of the ship motion model and the first-stage reinforcement learning ship collision avoidance decision model and designing a second-stage reward function with the proportion of dangerous tertiary actions to construct a second-stage safe reinforcement learning ship collision avoidance decision model for limiting the ship action space to ensure the safety of ship navigation when there are other ships or obstacles within a second threshold around the ship; training the second-stage safe reinforcement learning ship collision avoidance decision model to obtain a trained second-stage safe reinforcement learning ship collision avoidance decision model; when the ship and other ships or obstacles are within the first threshold, using the trained first-stage reinforcement learning ship collision avoidance decision model to make collision avoidance decisions during the ship's travel; when the ship and other ships or obstacles are within the second threshold, using the trained second-stage safe reinforcement learning ship collision avoidance decision model to make collision avoidance decisions during the ship's travel.

2. The ship collision avoidance decision method based on two-stage safe reinforcement learning according to claim 1, wherein: the first threshold condition refers to the maximum detection radius of the radar carried by the ship; the second threshold condition is determined by the collision risk value between the ship and other ships or obstacles.

3. The ship collision avoidance decision-making method based on two-stage security reinforcement learning according to claim 1, characterized in that: The international maritime collision avoidance rule compliance reward function is used to represent that if the ship complies with the rules, the ship is given a positive reward of 200, otherwise, the ship is given a negative reward of -200, and when multiple ships encounter, if another ship requires the ship to take responsibility for avoiding, the negative reward of the ship as a straight-ahead ship is cancelled. The expression of the international maritime collision avoidance rule compliance reward function is as follows: In the formula: COLREG reward : represents the reward function of the compliance with the International Regulations for Preventing Collisions at Sea; C success represents the number of ships that give way according to the International Regulations for Preventing Collisions at Sea; C stand-on represents the number of ships that make the request for giving way to the own ship in the situation of two ships meeting; A own_ship represents the action set of the own ship, in which the target heading is always the direction of the terminal point, and otherwise -1.

4. The ship collision avoidance decision-making method based on two-stage security reinforcement learning according to claim 1, characterized in that: The navigation economy reward function is used to define the angle difference between the ship's heading and the destination. The expression of the navigation economy reward function is as follows: economy reward =|C dest -C ownship |×α where: economy reward represents the sailing economy reward function; C dest is the direction of the destination relative to the ship; C ownship is the true heading of the ship; and α is the agent economy coefficient; 5. The ship collision avoidance decision-making method based on two-stage security reinforcement learning according to claim 1, characterized in that: The minimum encounter distance reward function is used to calculate the reward given by the minimum encounter distance between ships. The expression of the minimum encounter distance reward function is as follows: where: DCPA reward represents the minimum encounter distance reward function, C other_ship is the number of own ships in the environment.

6. The ship collision avoidance decision-making method based on two-stage security reinforcement learning according to claim 1, characterized in that: The first-stage reinforcement learning ship collision avoidance decision model comprises: a first-stage scene generation module for generating possible encounter scenes of the ship; a first-stage state recognition module for classifying another ship in the initial environment based on the relative bearing, distance, and heading of the ship to recognize the ship encounter scene; a first-stage reward function module for outputting reward values according to multiple reward items, i.e., the international maritime collision avoidance rule compliance reward function, the navigation economy reward function, and the minimum encounter distance reward function; a first-stage reward statistics module for statistically processing the reward values output by the first-stage reward function module; The first-stage reinforcement learning algorithm module is configured to output a decision on whether to update the current environment based on a reward value output by the first-stage reward function module, a scenario in which the ship may encounter output by the scenario generation module, and a current scenario. The first-stage update environment module is configured to update the environment in which the ship is running based on a decision output by the reinforcement learning algorithm module, a ship encounter scenario output by the state identification module, and a motion model of the ship.

7. The ship collision avoidance decision-making method based on two-stage security reinforcement learning according to claim 1, characterized in that: The second-stage safe reinforcement learning ship collision avoidance decision model comprises: The second-stage scenario generation module is configured to generate a scenario in which the ship may encounter. The second-stage state identification module is configured to classify the other ship in the initial environment based on a relative bearing, a distance, and a heading of the other ship relative to the ship, and identify a ship encounter scenario. The collision detection module is configured to calculate a collision possibility in a future time step. The three-level tree decision module is configured to predict a next three-step action based on the first-stage reinforcement learning ship collision avoidance decision model, calculate a three-level node that has a collision risk based on a three-level node as a statistical basis using the collision detection module, and mark the three-level node. The second-stage safe reinforcement learning reward function module is configured to give a reward to the first-level action based on a three-level node safety and a probability of each action output by the three-level tree decision module. The second-stage reward statistics module is configured to count the reward output by the second-stage safe reinforcement learning reward function module. The second-stage reinforcement learning algorithm module is configured to output a decision on whether to update the current space based on a reward value output by the second-stage safe reinforcement learning reward function module, a scenario in which the ship may encounter output by the scenario generation module, and an action output by the three-level tree decision module. The update environment module is configured to update the current ship travel environment based on a ship motion model, a ship encounter scenario output by the second-stage state identification module, and a decision on whether to update the current space output by the second-stage reinforcement learning algorithm module.

8. The ship collision avoidance decision-making method based on two-stage security reinforcement learning according to claim 1, characterized in that: The second-stage safe reinforcement learning reward function expression is as follows: In the formula, Reward represents a reward function value corresponding to the first-level action, β represents whether the three-level action has a collision risk, 1 if the three-level action has a collision risk, and 0 otherwise, A represents an event "a second-level action i occurs in this state", and B represents an event "a three-level action j occurs in this state".

9. The ship collision avoidance decision-making method based on two-stage security reinforcement learning according to claim 7, characterized in that: The three-level tree decision module calculates a three-level node that has a collision risk based on a three-level node as a statistical basis using the collision detection module as follows: A three-level decision tree model is constructed for the current scenario and is iteratively calculated. If a decision node is determined to have a collision, all child nodes of the decision node are regarded as collision nodes. If all the child nodes of a node are collision nodes, the node itself is also determined as collision, and the collision node is defined as a dangerous action; The dangerous action is deleted to update the ship action space and play a safety guarantee role.

10. The ship collision avoidance decision-making method based on two-stage security reinforcement learning according to claim 9, characterized in that: When the number of collision nodes in the three-level decision tree is greater than or equal to the threshold number of dangerous nodes, the trained first-stage reinforcement learning ship collision avoidance decision-making model is adopted to take collision avoidance actions, When the number of collision nodes in the three-level decision tree is less than the threshold number of dangerous nodes, the trained second-stage reinforcement learning ship collision avoidance decision-making model is adopted to limit the ship action space to ensure the safety of ship navigation.

Images