Ship collision avoidance decision-making method based on two-stage safety reinforcement learning

Abstract

The invention discloses a ship collision avoidance decision method based on two-stage safety reinforcement learning, and the method comprises the following steps: constructing a first-stage reinforcement learning ship collision avoidance decision model which is used for enabling a ship to take collision avoidance actions when the ship has other ships or obstacles within a first threshold value around the ship; a second-stage safety reinforcement learning ship collision avoidance decision model is constructed and used for limiting the ship action space to ensure the ship navigation safety when the ship has other ships or obstacles within a surrounding second threshold value; under a first threshold condition, adopting the trained first-stage reinforcement learning ship collision avoidance decision model to carry out collision avoidance decision in the ship running process; and when the ship and other ships or obstacles are in a second threshold condition, adopting the trained second-stage safety reinforcement learning ship collision avoidance decision model to carry out collision avoidance decision in the ship driving process. According to the method, the safety and economy of ship navigation are improved, and a safety guarantee is provided for a reinforcement learning collision avoidance algorithm.

Description

Technical Field

[0001] The invention belongs to the technical field of ship navigation and relates to a ship collision avoidance decision method based on two-stage safety reinforcement learning. Background Art

[0002] With the booming development of global maritime traffic, the number of ships is also increasing rapidly. This growth trend makes maritime navigation face greater challenges and risks. The harm caused by maritime accidents to the safety of life and property is also becoming increasingly prominent. Therefore, using intelligent technology to achieve autonomous collision avoidance decisions for ships, thereby reducing the impact of human factors on ship collision avoidance accidents, has become one of the hot topics widely concerned in the industry.

[0003] In the existing ship collision avoidance decision-making algorithms, the collision avoidance problem in high-density waters and complex waters is recognized as the difficulty in the ship collision avoidance problem. There are many ships in high-density waters, the navigation paths are complex and intertwined, and the collision avoidance requirements are extremely high, which makes traditional collision avoidance algorithms often perform poorly in such environments. Ordinary collision avoidance algorithms usually face two main problems: one is that due to insufficient modeling and prediction of the surrounding environment, the collision avoidance strategy is not comprehensive enough; the other is that in order to ensure safety, an overly conservative strategy is adopted, which excessively restricts the freedom of movement of the ship, significantly reducing the solution set and making the collision avoidance problem in the current scenario more difficult. Rule-oriented collision avoidance algorithms, such as traditional methods based on collision avoidance heading angles, may have difficulty in responding flexibly when dealing with high-density ship traffic, and may even cause ships to fall into a dilemma of being unable to effectively avoid collisions.

[0004] In contrast, reinforcement learning algorithms have shown unique advantages in dealing with collision avoidance problems in high-density and complex waters. Reinforcement learning can adaptively learn the best collision avoidance strategy through a large number of interactions with the environment. When faced with complex and changing navigation environments, it can solve challenges that traditional algorithms find difficult to cope with in a more flexible way. In addition, reinforcement learning algorithms do not rely on pre-designed rules, but gradually optimize collision avoidance behavior in a dynamic environment through a trial-and-error mechanism, which enables them to perform better in highly complex navigation scenarios.

[0005] However, the main drawback of the reinforcement learning collision avoidance algorithm is that it is essentially a "black box" model and lacks interpretability. The learning process of the algorithm is complex and it is difficult to clearly explain the specific decision-making basis, which may reduce its reliability in practical applications. Especially in critical safety areas such as ship collision avoidance, the transparency and verifiability of the decision-making process are crucial, and the uninterpretability of the reinforcement learning algorithm makes it difficult for users and regulators to trust the decision-making process. This is also one of the biggest challenges faced by reinforcement learning in the promotion and application of actual production environments.

[0006] Existing ship collision avoidance algorithms have made a lot of progress in recent years. However, despite the continuous advancement of technology, there are still some limitations, which make the existing ship collision avoidance algorithms face challenges in practical applications. The current mainstream collision avoidance algorithms can be divided into collision avoidance algorithms based on path planning and collision avoidance algorithms based on machine learning.

[0007] For collision avoidance algorithms based on path planning, such as the A* algorithm, speed barrier method, artificial potential field method, and heuristic algorithm, etc. The advantage of this type of algorithm is that it calculates the optimal collision avoidance path through a mathematical model and is interpretable. Although these optimization algorithms can find the optimal path under specific conditions, they also have several problems. First, the computational complexity is high. When multiple ships are avoiding collisions at the same time, the complexity of calculating the optimal path increases exponentially, making it difficult for the algorithm to meet real-time requirements in practical applications. In actual navigation, ships need to make decisions quickly, and the high computational requirements of the optimization algorithm may not be able to provide effective path planning in time. Secondly, these algorithms often ignore the economic efficiency of navigation. Most optimization algorithms focus on avoiding collisions, but ignore the economic efficiency of navigation, such as fuel consumption and time cost. The lack of comprehensive consideration of economy may lead to the algorithm being unable to balance safety and efficiency in practice, reducing the value of practical applications.

[0008] In recent years, the application of collision avoidance algorithms based on artificial intelligence, especially reinforcement learning collision avoidance algorithms, has gradually increased in the field of ship collision avoidance. Through reinforcement learning, the system can learn collision avoidance strategies through a large amount of training data. However, the existing reinforcement learning collision avoidance algorithms are all single reinforcement learning throughout the entire process, and the intelligent agents often cannot achieve both navigation economy and navigation safety. In addition, due to the special black box characteristics of reinforcement learning and the serious consequences caused by ship collisions, reinforcement learning collision avoidance algorithms are difficult to put into actual production activities.

[0009] A multi-dimensional comparative analysis of the two existing collision avoidance algorithms in terms of computational complexity, real-time responsiveness, interpretability, and safety assurance shows that the existing collision avoidance algorithms largely ignore the economic efficiency of navigation, causing autonomous surface ships equipped with the algorithms to always avoid the target ship too conservatively. Secondly, many algorithms assume that the ship can change its course or speed immediately, without fully considering the physical characteristics of the ship, such as inertia, turning curvature, etc., and have limited application to the ship motion model.

[0010] In summary, if the collision avoidance algorithm needs to be put into actual production application, the following problems need to be solved:

[0011] ① The collision avoidance algorithm needs to be able to handle complex sea areas where multiple ships may encounter each other;

[0012] ② For collision avoidance algorithms based on artificial intelligence, there needs to be a means to explain or ensure their safety;

[0013] ③ In the long term, when autonomous surface ships and manned ships co-exist at sea, the economic efficiency of ship navigation needs to be considered, and overly conservative actions should not be taken in encounters with other ships;

[0014] ④ When building a collision avoidance algorithm, it is necessary to take into account the actual navigation conditions as much as possible, including ship motion models, collision detection and other factors. Summary of the invention

[0015] In order to solve the above problems, the technical solution adopted by the present invention is: a ship collision avoidance decision method based on two-stage safety reinforcement learning, comprising the following steps:

[0016] Design ship motion model based on reinforcement learning scenario set and parameters;

[0017] Based on the ship motion model and combined with the reinforcement learning algorithm, we designed the reward function for compliance with the International Maritime Collision Avoidance Regulations, the navigation economy reward function, and the minimum encounter distance reward function, and constructed the first-stage reinforcement learning ship collision avoidance decision model, which is used for ships to take collision avoidance actions when there are other ships or obstacles within the first threshold around the ship.

[0018] The first-stage reinforcement learning ship collision avoidance decision model is trained to obtain a trained first-stage reinforcement learning ship collision avoidance decision model;

[0019] Based on the ship motion model and the action probability distribution of the first-stage reinforcement learning ship collision avoidance decision model, the proportion of each three-level action is determined and the proportion of dangerous three-level actions is used to design the reward function of the second stage, and the second-stage safety reinforcement learning ship collision avoidance decision model is constructed. It is used to limit the ship's action space to ensure the safety of the ship's navigation when there are other ships or obstacles within the second threshold around the ship.

[0020] The second-stage safety reinforcement learning ship collision avoidance decision model is trained to obtain a trained second-stage safety reinforcement learning ship collision avoidance decision model;

[0021] When the ship is in the first threshold condition with other ships or obstacles, the trained first-stage reinforcement learning ship collision avoidance decision model is used to make collision avoidance decisions during the ship's navigation;

[0022] When a ship is in collision with other ships or obstacles under the second threshold condition, the trained second-stage safety reinforcement learning ship collision avoidance decision model is used to make collision avoidance decisions during the ship's navigation.

[0023] Further: the first threshold condition refers to: the maximum detection radius of the radar carried by the ship;

[0024] The second threshold condition refers to: determining the collision risk value between the ship and other ships or obstacles.

[0025] Further: the reward function for complying with the International Maritime Collision Prevention Regulations is used to indicate that if the ship complies with the regulations, a positive reward of 200 is given to the ship, otherwise a negative reward of -200 is given. When multiple ships meet, if another ship requires the ship to avoid the responsibility, the negative reward of the ship as a direct-travel ship is cancelled;

[0026] The expression of the reward function for complying with the International Maritime Collision Prevention Regulations is as follows:

[0027]

[0028] Where: COLREG reward : indicates the reward function for compliance with the International Maritime Collision Prevention Regulations; C success Indicates the number of ships that make avoidance according to the International Regulations for Preventing Collisions at Sea; C stand-on Indicates the number of ships that requested to give way to the ship in the encounter situation; A own_ship Represents the action set of this ship. In this reward, the target heading is always 1 if the destination direction is the destination direction, otherwise it is -1.

[0029] Further: the navigation economy reward function is used to define the angle difference between the ship's heading and the end point;

[0030] The expression of the navigation economy reward function is as follows:

[0031]

[0032] Where: economy reward represents the navigation economy reward function; C dest is the direction of the end point relative to own ship; is the true heading of the ship; α is the economic coefficient of the intelligent agent;

[0033] Further: the minimum encounter distance reward function is used to calculate the reward given by calculating the minimum encounter distance between ships;

[0034] The expression of the minimum encounter distance reward function is as follows:

[0035]

[0036]

[0037] Where: DCPA reward represents the minimum encounter distance reward function, C other_ship is the number of other ships in the environment.

[0038] Further: the first stage reinforcement learning ship collision avoidance decision model includes:

[0039] The first stage scenario generation module: used to generate scenarios that the ship may encounter;

[0040] The first stage state recognition module: classifies other ships in the initial environment and identifies the ship encounter scenario based on their relative position, distance and heading to the ship.

[0041] The first stage reward function module is used to output reward values ​​according to multiple reward items, namely, the international maritime collision avoidance regulations compliance reward function, the navigation economy reward function, and the minimum encounter distance reward function;

[0042] The first-stage reward statistics module is used to count the reward values ​​output by the first-stage reward function module;

[0043] The first-stage reinforcement learning algorithm module is used to make a decision on whether to update the current environment based on the reward value output by the first-stage reward function module, the possible scenarios that the ship may encounter output by the scenario generation module, and the current scenario based on the output of the reinforcement learning algorithm;

[0044] The first stage updates the environment module: it is used to update the environment during the operation of the ship based on the decision output by the reinforcement learning algorithm module, the ship encounter scenario output by the state recognition module, and the ship's motion model.

[0045] Further: The second stage safety reinforcement learning ship collision avoidance decision model includes:

[0046] The second stage scenario generation module: used to generate scenarios that the ship may encounter;

[0047] The second stage state recognition module is used to classify other ships in the initial environment and identify the ship encounter scenario based on their relative position, distance and heading to the ship.

[0048] Collision detection module: used to calculate the possibility of collision in future time steps;

[0049] Three-level tree decision module: used to predict the actions of the next three steps based on the first-stage reinforcement learning ship collision avoidance decision model, and use the collision detection module to calculate and mark the third-level nodes with collision risks based on the third-level nodes as statistical basis; if a node is judged to be dangerous, all its child nodes are marked as having collision risks, and then the probability distribution of the action probability of the first-stage reinforcement model is used to obtain the probability of each node, and the third-level nodes are assigned the corresponding proportion; and the dangerous actions are deleted to play a safety guarantee role;

[0050] The reward function module of the second stage of safety reinforcement learning is used to reward the first-level action based on the safety and probability of the third-level nodes of each action output by the three-level tree decision module;

[0051] The second-stage reward statistics module is used to count the rewards output by the reward function module of the second-stage secure reinforcement learning;

[0052] The second-stage reinforcement learning algorithm module is used to make a decision on whether to update the current space based on the output of the reinforcement learning algorithm, based on the reward value output by the reward function module of the second-stage safety reinforcement learning, the scene that the ship may encounter output by the scene generation module, and the action output by the three-level tree decision module;

[0053] Update environment module: used to update the current ship driving environment based on the ship motion model, the ship encounter scenario output by the second stage state recognition and the decision of whether to update the current space output by the second stage reinforcement learning algorithm module.

[0054] Furthermore: The reward function expression of the second stage of security reinforcement learning is as follows:

[0055]

[0056] Where Reward represents the reward function value corresponding to the first-level action; β represents whether the third-level action has a collision risk, 1 if there is a collision risk, otherwise 0; Event A is "secondary action i occurs in this state"; Event B is "tertiary action j occurs in this state".

[0057] Further: the process of the three-level tree decision module using the three-level nodes as statistical basis and using the collision detection module to calculate the three-level nodes with collision risks is as follows:

[0058] Construct a three-level decision tree model for the current scenario and traverse the calculation; use the three-level nodes as the statistical basis to use the collision detection module to calculate the three-level nodes with collision risks and mark them

[0059] If a decision node is judged to be a collision, all its child nodes are considered collision nodes.

[0060] If all child nodes of a node are collision nodes, the node itself is also judged as collision, and the collision node is defined as a dangerous action;

[0061] Delete dangerous actions and update the ship's action space to ensure safety.

[0062] Furthermore: when the number of collision nodes in the three-level decision tree is ≥ the threshold number of dangerous nodes, the trained first-stage reinforcement learning ship collision avoidance decision model is used to take collision avoidance actions.

[0063] When the number of collision nodes in the three-level decision tree is less than the threshold number of dangerous nodes, the trained second-stage reinforcement learning ship collision avoidance decision model is used to limit the ship's action space to ensure the safety of ship navigation.

[0064] The present invention provides a ship collision avoidance decision-making method based on two-stage safety reinforcement learning, which has the following advantages:

[0065] Based on the two-stage reinforcement learning method, a number of modules are added to assist decision-making, and the decision space of the ship is dynamically restricted to ensure its safety. A method for building a ship collision avoidance algorithm is proposed.

[0066] This method models the collision avoidance process of the ship as a Markov decision process, and by designing different reward functions and action spaces for each stage, the intelligent agent can obtain effective collision avoidance decisions from the environment through a neural network (reinforcement learning algorithm). In addition, this method ensures the safety of decision-making through multi-layer decision trees and action space restrictions.

[0067] In the first stage, the method improves the navigation economy in non-high-risk situations through reinforcement learning, while in the second stage, the collision avoidance performance is significantly improved in complex waters or multi-ship encounter situations through safety decision induction and action space restriction. This provides a practical method for applying reinforcement learning collision avoidance algorithms to engineering practice.

[0068] The present invention achieves the following results by designing a two-stage safety reinforcement learning collision avoidance method: first, the method can ensure that ships can achieve efficient navigation that meets economic benefits under normal conditions; second, in high-risk scenarios, the algorithm uses a strict three-level tree decision module and collision detection module to build a multi-layer decision tree and action space constraints to eliminate potential dangerous paths and select safe and robust collision avoidance decisions for ships. This innovative dual reinforcement learning mechanism not only retains the flexible adaptability of reinforcement learning algorithms to scenarios, but also provides application safety and reliability support for such algorithms in the field of ship collision avoidance, providing a new solution to the problem of ship collision avoidance in high-density complex waters.

[0069] The present invention proposes a ship autonomous collision avoidance decision-making method based on reinforcement learning, which has the following beneficial effects.

[0070] First, autonomous learning and dynamic adaptability: This method can dynamically adjust the collision avoidance strategy through continuous interaction with complex dynamic environments, avoiding the limitations of relying on pre-designed complex rules or models, and is particularly suitable for navigation scenarios in high-density waters and complex waters. Through the experience accumulation of reinforcement learning, the intelligent agent can make the best choice when faced with complex nonlinear decision-making problems, thereby effectively solving the decision-making instability of traditional algorithms in complex environments.

[0071] Secondly, the balance of two-stage reinforcement learning: The present invention achieves a balance between economy and safety at different stages of navigation through two-stage reinforcement learning. The first stage focuses on optimizing navigation efficiency on the basis of complying with international maritime collision regulations, so that ships can flexibly perform their duties as give-way vessels. The second stage focuses on ensuring their own safety in high-risk scenarios, especially when other ships fail to fulfill their avoidance obligations. This dual strategy design ensures the collision avoidance capability of ships in complex waters, while also improving the compatibility of the method, which can automatically switch to the appropriate decision-making mode according to different scenarios.

[0072] Finally, the robustness of the second stage of safety reinforcement learning: reinforcement learning in this stage can not only ensure the safety of ship decision-making through safety decision induction and action space restriction, but also perform well in high-risk scenarios such as complex waters or entering and leaving ports. By introducing more navigation information, such as ship parameters, movement capabilities, water depth and terrain, and the intelligent algorithms of other ships, the robustness and practical applicability of collision avoidance decisions are further enhanced, providing solid technical support for the practical engineering application of reinforcement learning collision avoidance algorithms.

[0073] In summary, the invention has certain beneficial effects in the field of ship intelligent navigation technology. It improves the safety and economy of ship navigation, provides a safety guarantee for the reinforcement learning collision avoidance algorithm, and provides a research idea for the safety and reliability of the ship intelligent navigation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0075] Figure 1 It is a flowchart of a ship collision avoidance decision-making method based on two-stage safety reinforcement learning;

[0076] Figure 2 It is the design flow chart of the first stage reinforcement learning ship collision avoidance decision model;

[0077] Figure 3 It is the design flow chart of the second stage safety reinforcement learning ship collision avoidance decision model;

[0078] Figure 4 It is a quantitative schematic diagram of the encounter scenario of the International Regulations for Preventing Collisions at Sea;

[0079] Figure 5This is a schematic diagram of the ship size;

[0080] Figure 6 is a decision tree model;

[0081] Figure 7 The model performance in the single-ship encounter scenario, where (a) is a single-ship encounter, (b) is a single-ship right-crossing with a small angle, (c) is a single-ship right-crossing with a medium angle, and (d) is a single-ship crossing with a large angle.

[0082] Figure 8 The model performance in the multi-ship encounter scenario, where (a) is a four-ship encounter situation and (b) is a multi-ship encounter situation. DETAILED DESCRIPTION

[0083] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0084] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] Figure 1 It is a flowchart of a ship collision avoidance decision-making method based on two-stage safety reinforcement learning;

[0086] A ship collision avoidance decision method based on two-stage safety reinforcement learning is divided into two stages. The first stage of reinforcement learning focuses on the navigation economy and compliance with the rules of the ship, aiming to enable the ship to efficiently complete the navigation task under the premise of following the rules. The second stage of safety reinforcement learning focuses on the safety of navigation and further ensures the safety of its decision-making by limiting the action space. Therefore, it is necessary to set a mode transition condition related to safety to determine when to adopt which decision-making method.

[0087] The method comprises the following steps:

[0088] S1: Design a ship motion model based on reinforcement learning scenario sets and parameters;

[0089] S2: Based on the ship motion model and combined with the reinforcement learning algorithm, the international maritime collision avoidance regulations compliance reward function, navigation economy reward function, and minimum encounter distance reward function are designed to construct the first-stage reinforcement learning ship collision avoidance decision model, which is used for the ship to take collision avoidance actions when there are other ships or obstacles within the first threshold around the ship; in this process, the economy of navigation must be considered as much as possible to avoid overly conservative collision avoidance decisions; and weights are allocated for each action in the second stage of safety reinforcement learning;

[0090] S3: training the first-stage reinforcement learning ship collision avoidance decision model to obtain a trained first-stage reinforcement learning ship collision avoidance decision model;

[0091] S4: Based on the ship motion model and the action probability distribution of the first-stage reinforcement learning ship collision avoidance decision model, the proportion of each three-level action is determined and the proportion of dangerous three-level actions is used to design the reward function of the second stage, and the second-stage safety reinforcement learning ship collision avoidance decision model is constructed to limit the ship's action space to ensure the safety of the ship's navigation when there are other ships or obstacles within the second threshold around the ship;

[0092] S5: training the second-stage safety reinforcement learning ship collision avoidance decision model to obtain a trained second-stage safety reinforcement learning ship collision avoidance decision model;

[0093] S6: When the ship is in the first threshold condition with other ships or obstacles, the trained first-stage reinforcement learning ship collision avoidance decision model is used to make collision avoidance decisions during the ship's navigation;

[0094] When a ship is in collision with other ships or obstacles under the second threshold condition, the trained second-stage safety reinforcement learning ship collision avoidance decision model is used to make collision avoidance decisions during the ship's navigation.

[0095] The steps S1 / S2 / S3 / S4 / S5 / S6 are performed sequentially;

[0096] Further: the first threshold condition refers to: the maximum detection radius of the radar carried by the ship; the maximum detection radius in this embodiment is 6 nautical miles;

[0097] The second threshold condition refers to: determining the collision risk value between the ship and other ships or obstacles.

[0098] Figure 2 It is the design flow chart of the first stage reinforcement learning ship collision avoidance decision model;

[0099] The first stage reinforcement learning ship collision avoidance decision model includes:

[0100] The first stage scenario generation module: used to generate scenarios that the ship may encounter;

[0101] The first stage state recognition module: classifies other ships in the initial environment and identifies the ship encounter scenario based on their relative position, distance and heading to the ship.

[0102] The first stage reward function module is used to output reward values ​​according to multiple reward items, namely, the international maritime collision avoidance regulations compliance reward function, the navigation economy reward function, and the minimum encounter distance reward function;

[0103] The first-stage reward statistics module is used to count the reward values ​​output by the first-stage reward function module;

[0104] The first-stage reinforcement learning algorithm module is used to make a decision on whether to update the current environment based on the reward value output by the first-stage reward function module, the possible scenarios that the ship may encounter output by the scenario generation module, and the current scenario based on the output of the reinforcement learning algorithm;

[0105] The first stage updates the environment module: it is used to update the environment during the operation of the ship based on the decision output by the reinforcement learning algorithm module, the ship encounter scenario output by the state recognition module, and the ship's motion model.

[0106] The process of training the first-stage reinforcement learning ship collision avoidance decision model;

[0107] During the reinforcement learning training process, the scene generation module, state recognition module, system motion model and reward function are integrated into a unified environment and connected to the reinforcement learning algorithm for training.

[0108] First, the environment is constructed and initialized, integrating the above modules to form a complete training platform. The environment can generate a variety of training scenarios in real time and feedback the system status information. At the beginning of each training, the environment has a certain random initialization ability to create new scenarios and states, providing a variable and challenging training background for the agent.

[0109] Next, the agent is trained. At each time step, the agent selects an action based on the current state according to the selected reinforcement learning algorithm. Whether it is a policy-based algorithm (such as PPO, TRPO) or a value-based algorithm (such as DQN, DDPG), the agent will use the current policy or value function to generate actions, which enables the agent to explore different decision paths during training. Subsequently, the selected action will be applied to the environment, causing the system state to change. The environment will update the state according to the execution result and calculate the reward value to reflect the performance of the agent in that state.

[0110] During the training process, the agent continuously optimizes its decision-making strategy by collecting experience of state, action, and reward. For policy optimization algorithms, such as PPO or A3C, these experiences are used to update the policy. For value function optimization algorithms, such as DQN, these experiences are used to update the Q-value function. Reinforcement learning algorithms use different update methods to ensure that the agent gradually improves its strategy and eventually learns to make the best decision in different scenarios.

[0111] Finally, during the training process, the reward statistics module will regularly monitor the agent's cumulative rewards. As the training progresses, the agent's cumulative rewards should gradually increase, indicating that its strategy is being optimized. By calculating the magnitude of the change in the strategy or value function, we can determine whether the training has converged. If the strategy or value function changes very little after multiple updates, it means that the agent has found a relatively stable decision-making strategy, and it can be judged that the training has achieved the desired effect.

[0112] In the first stage of reinforcement learning, three reward functions are designed:

[0113] Regarding the rewards for compliance with the International Regulations for Preventing Collisions at Sea (incident rewards):

[0114] In the International Regulations for Preventing Collisions at Sea, Article 13 requires that when two ships are in an overtaking situation, the overtaking ship should give way to the overtaken ship; Article 14 requires that when two ships are in a meeting situation, they should each turn right so that each passes by the other ship's port side; Article 15 requires that when two ships are in a crossing situation, the other ship on the starboard side of the ship should give way to the other ship and, if the circumstances permit, avoid crossing in front of the other ship; in addition, if the ship is a straight-line ship, it should also fulfill the obligation to maintain speed and direction.

[0115] According to the description of the International Regulations for Preventing Collisions at Sea, the International Regulations for Preventing Collisions at Sea compliance reward function is used to indicate that if the ship complies with the regulations, a positive reward of 200 will be given to the ship, otherwise a negative reward of -200 will be given. When multiple ships meet, if another ship requires the ship to avoid the responsibility, the negative reward of the ship as a direct-travel ship will be cancelled;

[0116] The expression of the reward function for complying with the International Maritime Collision Prevention Regulations is as follows:

[0117]

[0118] Where: COLREG reward : indicates the reward function for compliance with the International Maritime Collision Prevention Regulations; C success Indicates the number of ships that make avoidance according to the International Regulations for Preventing Collisions at Sea; C stand-on Indicates the number of ships that requested to give way to the ship in the encounter situation; A own_shipRepresents the action set of this ship. In this reward, the target heading is always 1 if the destination direction is the destination direction, otherwise it is -1.

[0119] About the navigation economy reward (step-by-step reward) function;

[0120] The navigation economy reward function is used to define the angular difference between the ship's heading and the end point; the weight of this reward directly determines the agent's emphasis on navigation economy. The larger the weight, the smaller the detour distance. Conversely, the smaller the weight, the more emphasis is placed on navigation safety.

[0121] The expression of the navigation economy reward function is as follows:

[0122]

[0123] Where: economy reward represents the navigation economy reward function; C dest is the direction of the end point relative to own ship; is the true heading of the ship; α is the economic coefficient of the intelligent agent; here α=0.0035.

[0124] About the minimum encounter distance reward function;

[0125] The minimum encounter distance reward function is used to calculate the reward given for the minimum encounter distance between ships; in order to avoid an emergency situation caused by the two ships being too close, it is also necessary to be careful not to make the reward too large.

[0126] The expression of the minimum encounter distance reward function is as follows:

[0127]

[0128]

[0129] Where: DCPA reward represents the minimum encounter distance reward function, C other_ship is the number of other ships in the environment.

[0130] Figure 3 It is the design flow chart of the second stage safety reinforcement learning ship collision avoidance decision model;

[0131] The second stage safety reinforcement learning ship collision avoidance decision model includes:

[0132] The second stage scenario generation module: used to generate scenarios that the ship may encounter;

[0133] The second stage state recognition module is used to classify other ships in the initial environment and identify the ship encounter scenario based on their relative position, distance and heading to the ship.

[0134] Collision detection module: used to calculate the possibility of collision in future time steps;

[0135] Three-level tree decision module: used to predict the actions of the next three steps based on the first-stage reinforcement learning ship collision avoidance decision model, and use the collision detection module to calculate and mark the third-level nodes with collision risks based on the third-level nodes as statistical basis; if a node is judged to be dangerous, all its child nodes are marked as having collision risks, and then the probability distribution of the action probability of the first-stage reinforcement model is used to obtain the probability of each node, and the third-level nodes are assigned the corresponding proportion; and the dangerous actions are deleted to play a safety guarantee role;

[0136] The reward function module of the second stage of safety reinforcement learning is used to reward the first-level action based on the safety and probability of the third-level nodes of each action output by the three-level tree decision module;

[0137] The second-stage reward statistics module is used to count the rewards output by the reward function module of the second-stage secure reinforcement learning;

[0138] The second-stage reinforcement learning algorithm module is used to make a decision on whether to update the current space based on the output of the reinforcement learning algorithm, based on the reward value output by the reward function module of the second-stage safety reinforcement learning, the scene that the ship may encounter output by the scene generation module, and the action output by the three-level tree decision module;

[0139] Update environment module: used to update the current ship driving environment based on the ship motion model, the ship encounter scenario output by the second stage state recognition and the decision of whether to update the current space output by the second stage reinforcement learning algorithm module.

[0140] The second-stage reward function is designed as a safety reward function based on a decision tree:

[0141] The second stage reward function expression is as follows:

[0142]

[0143] Where Reward represents the reward function value corresponding to the first-level action; β represents whether the third-level action has a collision risk, 1 if there is a collision risk, otherwise 0; Event A is "secondary action i occurs in this state"; Event B is "tertiary action j occurs in this state".

[0144] The process of the three-level tree decision module using the three-level nodes as statistical basis and the collision detection module to calculate the three-level nodes with collision risks is as follows:

[0145] Construct a three-level decision tree model for the current scenario and traverse the calculation; use the three-level nodes as the statistical basis to use the collision detection module to calculate the three-level nodes with collision risks and mark them

[0146] If a decision node is judged to be a collision, all its child nodes are considered collision nodes.

[0147] If all child nodes of a node are collision nodes, the node itself is also judged as collision, and the collision node is defined as a dangerous action;

[0148] Delete dangerous actions and update the ship's action space to ensure safety;

[0149] When the number of collision nodes in the three-level decision tree is ≥ the threshold number of dangerous nodes, the trained first-stage reinforcement learning ship collision avoidance decision model is used to take collision avoidance actions.

[0150] When the number of collision nodes in the three-level decision tree is less than the threshold number of dangerous nodes, the trained second-stage reinforcement learning ship collision avoidance decision model is used to limit the ship's action space to ensure the safety of ship navigation.

[0151] The process of training the second stage safety reinforcement learning ship collision avoidance decision model is as follows:

[0152] The scenario generation module, state recognition module, ship motion model, three-level decision tree and second-stage reward function are integrated into the second-stage safety reinforcement learning environment, and the environment is connected to the PPO algorithm for safety reinforcement learning training.

[0153] During the reinforcement learning training process, the scene generation module, state recognition module, ship motion model, three-level decision tree module and the second-stage safety reinforcement learning reward function are integrated into a unified environment and connected to the reinforcement learning algorithm for training.

[0154] First, the environment is constructed and initialized, integrating the above modules to form a complete training platform. The environment can generate a variety of training scenarios in real time and feedback the system status information. At the beginning of each training, the environment has a certain random initialization ability to create new scenarios and states, providing a variable and challenging training background for the agent.

[0155] Next, the agent is trained. At each time step, the agent selects an action based on the current state according to the selected reinforcement learning algorithm. Whether it is a policy-based algorithm (such as PPO, TRPO) or a value-based algorithm (such as DQN, DDPG), the agent will use the current policy or value function to generate actions, which enables the agent to explore different decision paths during training. Subsequently, the selected action will be applied to the environment, causing the system state to change. The environment will update the state according to the execution result and calculate the reward value to reflect the performance of the agent in that state.

[0156] During the training process, the agent continuously optimizes its decision-making strategy by collecting experience of state, action, and reward. For policy optimization algorithms, such as PPO or A3C, these experiences are used to update the policy. For value function optimization algorithms, such as DQN, these experiences are used to update the Q-value function. Reinforcement learning algorithms use different update methods to ensure that the agent gradually improves its strategy and eventually learns to make the best decision in different scenarios.

[0157] In addition, through the prediction process of the three future time steps based on the current moment by the three-level decision tree, the feasible range of the action space at the current moment is returned, and the action space is limited to a safe range.

[0158] Finally, during the training process, the reward statistics module will regularly monitor the agent's cumulative rewards. As the training progresses, the agent's cumulative rewards should gradually increase, indicating that its strategy is being optimized. By calculating the magnitude of the change in the strategy or value function, we can determine whether the training has converged. If the strategy or value function changes very little after multiple updates, it means that the agent has found a relatively stable decision-making strategy, and it can be judged that the training has achieved the desired effect.

[0159] Through the above two-stage reinforcement learning training process, the actual application of the intelligent agent in ship collision avoidance is finally realized. When any reinforcement learning call conditions are not met, the ship sails normally along the established route; if the first stage conditions are met but the second stage conditions are not met, the first stage decision is called; if vice versa. When both stage call conditions are met at the same time, the second stage safety decision is prioritized.

[0160] Embodiment 1:

[0161] A ship collision avoidance decision method based on two-stage safety reinforcement learning comprises the following steps:

[0162] S1: Design a ship motion model based on reinforcement learning scenario sets and parameters;

[0163] For the first stage, 1 to 5 random companion ships are defined, with random angles relative to the own ship, distances within the range of 4-6 nautical miles, and headings within the range of ±20 degrees facing the heading of the own ship. It is assumed that the other ships are non-behaving ships that maintain speed and direction.

[0164] For the second stage, 1 to 5 random companion ships are defined, with random angles relative to the own ship, distances within the range of 1-2 nautical miles, and headings within the range of ±10 degrees facing the heading of the own ship. It is assumed that the other ships are non-behaving ships that maintain speed and direction.

[0165] Define the ship's speed as 10 knots and the dimensions as Figure 5 The action space of reinforcement learning is the expected course within this time step. The state space is the position, distance, course of all other ships in the environment relative to the ship, and the type of encounter situation with the ship. The time step for environment update and decision-making is 80s.

[0166] The ship motion model uses the Nomoto model based on the fourth-order Runge-Kutta numerical integration method to simulate the motion process of the ship, including the evolution of position, heading, speed and other states.

[0167] The ship motion model updates the state according to the following formula:

[0168]

[0169]

[0170]

[0171] Where: x is the x-coordinate of the ship in the plane; y is the y-coordinate of the ship in the plane; is the heading angle of the ship (the angle with the x-axis); u is the longitudinal velocity of the ship; v is the lateral velocity of the ship; r is the angular velocity of the ship.

[0172] According to the dynamic equations and PID control equations of the Nomoto model, the lateral velocity v and angular velocity r of the ship are controlled by the following differential equations:

[0173]

[0174] δ∈[δ min , δ max ]

[0175] Where: a 11 , a 12 , a 21 , a 22 , b 11 , b 21It is the motion parameter of the ship, which describes the influence of the damping and inertia effects of the ship on the lateral velocity and the turning angular velocity; is the target heading; K p is the proportional gain coefficient; K d is the differential gain coefficient; δ min and δ max They represent the minimum and maximum values ​​of the rudder angle that can be input by the ship respectively.

[0176] The model uses the fourth-order Runge-Kutta method to approximate the differential equation. This numerical method can provide higher accuracy and ensure the stability and accuracy of the ship motion model. The update steps are as follows:

[0177] k1=h·f(t n ,y n )

[0178]

[0179] k4=h·f(t n +h,y n +k3)

[0180]

[0181] Where: k1, k2, k3, k4 represent the estimated values ​​of the four slopes in the Runge-Kutta method; h is the time step; y n is the current state of the ship, including variables such as position, heading angle, speed, and rudder angle; t n is the current time; f is the state update function of the system;

[0182] S2: Based on the ship motion model and combined with the reinforcement learning algorithm, we designed the reward function for compliance with the International Maritime Collision Avoidance Regulations, the navigation economy reward function, and the minimum encounter distance reward function, and constructed the first-stage reinforcement learning ship collision avoidance decision model, which is used for the ship to take collision avoidance actions when there are other ships or obstacles within the first threshold around the ship. In this process, the economy of navigation must be considered as much as possible to avoid overly conservative collision avoidance decisions; and weights are allocated for each action in the second-stage safety reinforcement learning;

[0183] S3: Train the first-stage reinforcement learning ship collision avoidance decision model to obtain a trained first-stage reinforcement learning ship collision avoidance decision model; in the first-stage reinforcement learning training, if there are no other obstacles and ships within 6 nautical miles around the ship or it has not ended after 200 time steps, it means that the current training round is over, so as to obtain the probability distribution of the agent action output of the whole process.

[0184] S4: Based on the ship motion model and the action probability distribution of the first-stage reinforcement learning ship collision avoidance decision model, the proportion of each three-level action is determined and the proportion of dangerous three-level actions is used to design the reward function of the second stage, and the second-stage safety reinforcement learning ship collision avoidance decision model is constructed to limit the ship's action space to ensure the safety of the ship's navigation when there are other ships or obstacles within the second threshold around the ship;

[0185] The three-level decision tree structure used in the second stage safety reinforcement learning ship collision avoidance decision model is as follows: Figure 6 As shown;

[0186] For each action, a decision tree with three depths is established. According to the action space, each node has 72 sub-nodes, with a total of three layers.

[0187] For the second-level and third-level nodes, given the previous action, their specific scenarios and state spaces should be updated based on this action, and their action probabilities should be obtained from the first-stage reinforcement learning model.

[0188] The construction process of the reward function of the second stage of secure reinforcement learning is as follows:

[0189] First, a three-level decision tree model is constructed for the current scenario to predict the actions of the next three steps. The third-level nodes are used as statistical basis to calculate and mark the third-level nodes with collision risks using the collision detection module. In this process, if a node is judged to be dangerous, all its child nodes are marked as having collision risks. The probability distribution of the action of the first-stage reinforcement model is then used to obtain the probability of each node and assign the corresponding weight to the third-level nodes. Finally, based on the safety and probability of the third-level nodes of each action, rewards are given to the first-level actions;

[0190] Secondly, if a first-level action is judged to have a collision risk or the sum of the collision avoidance probabilities of the sub-nodes of the first-level action is 1, this first-level action will be temporarily removed from the agent action space to ensure the safety of the safety reinforcement learning ship collision avoidance decision-making algorithm.

[0191] In the second phase of safety reinforcement learning, if there are no other obstacles and ships or collision avoidance accidents within 2 nautical miles around the ship, it means that the current training round ends. This mechanism is designed to ensure that when the ship is in a non-urgent state and no further decision is required, the current task can be effectively terminated, saving computing resources and improving training efficiency.

[0192] S5: training the second-stage safety reinforcement learning ship collision avoidance decision model to obtain a trained second-stage safety reinforcement learning ship collision avoidance decision model;

[0193] S6: Integrate and test the two-stage secure reinforcement learning model:

[0194] First, an initial scenario is defined, where the ship position and heading are random. It should be noted that the number of other ships within the limited range of the two-stage reinforcement learning cannot exceed the dimension of the observation space.

[0195] When there are no other obstacles or ships within 6 nautical miles around the ship, the ship will sail towards the target course; when there are other ships or obstacles within 6 nautical miles around, the ship will start the first stage of reinforcement learning and take collision avoidance actions; when there are other ships or obstacles within 2 nautical miles around, the ship's two-stage reinforcement learning model will be fully activated. At this time, if the number of dangerous nodes in the decision tree is less than 10%, the collision avoidance decision of the first stage of reinforcement learning shall prevail, otherwise the second stage of safety reinforcement learning shall prevail. At this time, no matter which model is used, a three-layer decision tree should be constructed and the action space should be restricted to ensure navigation safety.

[0196] For single-ship collision avoidance scenarios, the algorithm can implement economical and rule-compliant collision avoidance strategies, such as Figure 7 As shown, (a) is a single ship encounter situation, (b) is a single ship small angle right fork situation, (c) is a single ship medium angle right cross situation, (d) is a single ship large angle cross situation;

[0197] The specific parameters of the scenario are shown in Table 1:

[0198] Scenario Other ship's horizontal coordinate (m) Other ship's vertical coordinate (m) Other ship's heading (deg) Encounter Type (a) 0 11110 Due South Encounter (b) 2520 10819 14°S Right Cross (c) 8290 7396 12°W by S Right Cross (d) 10932 1972 40° west by north Right Cross

[0199] When the number of surrounding ships increases, the model can demonstrate rule-based safety characteristics and maintain safe navigation in high-density and complex waters. Figure 8 As shown, (a) is a situation where four ships meet, and (b) is a situation where multiple ships meet.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship collision avoidance decision method based on two-stage safety reinforcement learning, characterized by: The following steps are involved: Design ship motion model based on reinforcement learning scenario set and parameters; Based on the ship motion model and combined with the reinforcement learning algorithm, we designed the reward function for compliance with the International Maritime Collision Avoidance Regulations, the navigation economy reward function, and the minimum encounter distance reward function, and constructed the first-stage reinforcement learning ship collision avoidance decision model, which is used for ships to take collision avoidance actions when there are other ships or obstacles within the first threshold around the ship. The first-stage reinforcement learning ship collision avoidance decision model is trained to obtain a trained first-stage reinforcement learning ship collision avoidance decision model; Based on the ship motion model and the action probability distribution of the first-stage reinforcement learning ship collision avoidance decision model, the proportion of each three-level action is determined and the proportion of dangerous three-level actions is used to design the reward function of the second stage, and the second-stage safety reinforcement learning ship collision avoidance decision model is constructed. It is used to limit the ship's action space to ensure the safety of the ship's navigation when there are other ships or obstacles within the second threshold around the ship. The second-stage safety reinforcement learning ship collision avoidance decision model is trained to obtain a trained second-stage safety reinforcement learning ship collision avoidance decision model; When the ship is in the first threshold condition with other ships or obstacles, the trained first-stage reinforcement learning ship collision avoidance decision model is used to make collision avoidance decisions during the ship's navigation; When a ship is in collision with other ships or obstacles under the second threshold condition, the trained second-stage safety reinforcement learning ship collision avoidance decision model is used to make collision avoidance decisions during the ship's navigation.

2. The ship collision avoidance decision method based on two-stage safety reinforcement learning according to claim 1 is characterized in that: The first threshold condition refers to: the maximum detection radius of the radar carried by the ship; The second threshold condition refers to: determining the collision risk value between the ship and other ships or obstacles.

3. The ship collision avoidance decision method based on two-stage safety reinforcement learning according to claim 1 is characterized in that: The reward function for complying with the International Regulations for Preventing Collisions at Sea is used to indicate that if the ship complies with the regulations, a positive reward of 200 will be given to the ship, otherwise a negative reward of -200 will be given. When multiple ships meet, if another ship requires the ship to avoid the responsibility, the negative reward of the ship as a direct-travel ship will be cancelled; The expression of the reward function for complying with the International Maritime Collision Prevention Regulations is as follows: Where: COLREG reward : indicates the reward function for compliance with the International Maritime Collision Prevention Regulations; C success Indicates the number of ships that make avoidance according to the International Regulations for Preventing Collisions at Sea; C stand-on Indicates the number of ships that requested to give way to the ship in the encounter situation; A own_ship Represents the action set of this ship. In this reward, the target heading is always 1 if the destination direction is the destination direction, otherwise it is -1.

4. The ship collision avoidance decision method based on two-stage safety reinforcement learning according to claim 1 is characterized in that: Said navigation economy reward function is used to define the angle difference between the ship's heading and the end point; The expression of the navigation economy reward function is as follows: economy reward =-|C dest -C ownship |×α Where: economy reward represents the navigation economy reward function; C dest is the direction of the end point relative to own ship; C ownship is the true heading of the ship; α is the economic coefficient of the intelligent agent.

5. The ship collision avoidance decision-making method based on two-stage safety reinforcement learning according to claim 1 is characterized in that: The minimum encounter distance reward function is used to calculate the reward given by the minimum encounter distance between ships; The expression of the minimum encounter distance reward function is as follows: Where: DCPA reward represents the minimum encounter distance reward function, C other_ship is the number of other ships in the environment.

6. The ship collision avoidance decision method based on two-stage safety reinforcement learning according to claim 1 is characterized in that: The first stage reinforcement learning ship collision avoidance decision model includes: The first stage scenario generation module: used to generate scenarios that the ship may encounter; The first stage state recognition module: classifies other ships in the initial environment and identifies the ship encounter scenario based on their relative position, distance and heading to the ship. The first stage reward function module is used to output reward values ​​according to multiple reward items, namely, the international maritime collision avoidance regulations compliance reward function, the navigation economy reward function, and the minimum encounter distance reward function; The first-stage reward statistics module is used to count the reward values ​​output by the first-stage reward function module; The first-stage reinforcement learning algorithm module is used to make a decision on whether to update the current environment based on the reward value output by the first-stage reward function module, the possible scenarios that the ship may encounter output by the scenario generation module, and the current scenario based on the output of the reinforcement learning algorithm; The first stage updates the environment module: it is used to update the environment during the operation of the ship based on the decision output by the reinforcement learning algorithm module, the ship encounter scenario output by the state recognition module, and the ship's motion model.

7. The ship collision avoidance decision-making method based on two-stage safety reinforcement learning according to claim 1 is characterized in that: The second stage safety reinforcement learning ship collision avoidance decision model includes: The second stage scenario generation module: used to generate scenarios that the ship may encounter; The second stage state recognition module is used to classify other ships in the initial environment and identify the ship encounter scenario based on their relative position, distance and heading to the ship. Collision detection module: used to calculate the possibility of collision in future time steps; Three-level tree decision module: used to predict the actions of the next three steps based on the first-stage reinforcement learning ship collision avoidance decision model, and use the collision detection module to calculate and mark the third-level nodes with collision risks based on the third-level nodes as statistical basis; if a node is judged to be dangerous, all its child nodes are marked as having collision risks, and then the probability distribution of the action probability of the first-stage reinforcement model is used to obtain the probability of each node, and the third-level nodes are assigned the corresponding proportion; and the dangerous actions are deleted to play a safety guarantee role; The reward function module of the second stage of safety reinforcement learning is used to reward the first-level action based on the safety and probability of the third-level nodes of each action output by the three-level tree decision module; The second-stage reward statistics module is used to count the rewards output by the reward function module of the second-stage secure reinforcement learning; The second-stage reinforcement learning algorithm module is used to make a decision on whether to update the current space based on the output of the reinforcement learning algorithm, based on the reward value output by the reward function module of the second-stage safety reinforcement learning, the scene that the ship may encounter output by the scene generation module, and the action output by the three-level tree decision module; Update environment module: used to update the current ship driving environment based on the ship motion model, the ship encounter scenario output by the second stage state recognition and the decision of whether to update the current space output by the second stage reinforcement learning algorithm module.

8. The ship collision avoidance decision-making method based on two-stage safety reinforcement learning according to claim 1 is characterized in that: The reward function expression of the second stage of secure reinforcement learning is as follows: Where Reward represents the reward function value corresponding to the first-level action; β represents whether the third-level action has a collision risk, which is 1 if there is a collision risk, otherwise it is 0; Event A is "secondary action i occurs in this state"; Event B is "tertiary action j occurs in this state".

9. The ship collision avoidance decision method based on two-stage safety reinforcement learning according to claim 7 is characterized in that: The process of the three-level tree decision module using the three-level nodes as statistical basis and the collision detection module to calculate the three-level nodes with collision risks is as follows: Construct a three-level decision tree model for the current scenario and traverse the calculation; use the three-level nodes as the statistical basis to use the collision detection module to calculate the three-level nodes with collision risks and mark them If a decision node is judged to be a collision, all its child nodes are considered collision nodes. If all child nodes of a node are collision nodes, the node itself is also judged as collision, and the collision node is defined as a dangerous action; Delete dangerous actions and update the ship's action space to ensure safety.

10. A ship collision avoidance decision method based on two-stage safety reinforcement learning according to claim 9, characterized in that: When the number of collision nodes in the three-level decision tree is ≥ the threshold number of dangerous nodes, the trained first-stage reinforcement learning ship collision avoidance decision model is used to take collision avoidance actions. When the number of collision nodes in the three-level decision tree is less than the threshold number of dangerous nodes, the trained second-stage reinforcement learning ship collision avoidance decision model is used to limit the ship's action space to ensure the safety of ship navigation.

Images