Automatic collision avoidance methods, devices, equipment and storage media for ships

Abstract

This invention belongs to the field of ship automation control and provides a method, device, equipment, and storage medium for automatic collision avoidance of ships. The method includes: constructing a collision avoidance model based on deep reinforcement learning training to formulate avoidance decisions; when a grid sensor detects other ships, acquiring the state information of the other ships and calculating the collision risk area based on the state information; when the collision risk area overlaps with the sensing area of ​​the grid sensor, determining whether the ship is a avoidant or a hold-off ship; if it is a hold-off ship, maintaining its course and speed; if it is an avoidant ship, calculating an avoidance decision; after executing the avoidance decision, updating the state information of the ship and other ships in the environmental information, repeating the process until the distance between the ship and other ships is greater than a threshold or less than a threshold and there is no collision risk; this invention aims to solve the problems of traditional ship collision avoidance technology being unsuitable for complex sea conditions and having high computational complexity.

Description

Technical Field

[0001] This invention relates to the field of ship automation control, and in particular to a method, apparatus, equipment and storage medium for automatic collision avoidance of ships. Background Technology

[0002] With the continuous increase in global maritime traffic, sea traffic is becoming increasingly congested. According to statistics, 89% to 95% of maritime accidents are caused by human factors. This highlights the vulnerability of crew members to errors in decision-making in complex navigation environments. Therefore, reducing maritime accidents, especially through automated collision avoidance operations, has become a key research direction in the field of marine engineering. Traditional model-based collision avoidance methods, such as artificial potential field methods, velocity barrier methods, A* algorithms, and particle swarm optimization algorithms, while performing well in specific scenarios, have limitations when dealing with complex and ever-changing real-world sea conditions. These methods are not only complex in modeling and computationally expensive, but also lack self-learning capabilities, making them difficult to adapt to dynamically changing environments. Summary of the Invention

[0003] In view of the above technical problems, the present invention provides a method, device, equipment and storage medium for automatic collision avoidance of ships, which aims to solve the problems that traditional ship collision avoidance technology is difficult to adapt to complex sea conditions and has high computational complexity.

[0004] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0005] According to one aspect of the present invention, an automatic collision avoidance method for ships is disclosed, the method comprising:

[0006] A collision avoidance model is constructed based on deep reinforcement learning to make avoidance decisions. The collision avoidance model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor used to detect information about the ship itself and its surrounding environment as input to the collision avoidance model. The decision layer includes a Q-Learning algorithm used to determine the optimal avoidance decision based on the input from the input layer. The action layer is used to adjust the ship's speed and heading based on the avoidance decision. The reward layer updates the output of the decision layer and includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is the reward for the ship heading toward the target point. The heading deviation reward is the reward for the ship heading toward the target point. The collision avoidance reward is the reward for the ship avoiding a collision. The rule reward is the reward for the ship turning right to avoid a collision.

[0007] When the grid sensor detects other vessels, it acquires the status information of the other vessels, including speed and heading, and calculates the collision risk area based on the status information.

[0008] When the collision risk area overlaps with the sensing area of ​​the grid sensor, it determines whether it is a avoidance vessel or a hold-off vessel. If it is a hold-off vessel, it maintains its course and speed. If it is an avoidance vessel, it inputs the status information of other vessels into the collision avoidance model to calculate the avoidance decision. After the avoidance decision is executed at the decision layer, the status information of itself and other vessels is updated into the environmental information. This process is repeated until the distance between the vessel and other vessels is greater than a threshold or less than a threshold and there is no collision risk.

[0009] Furthermore, when calculating the area where the collision risk occurs, the calculation is as follows:

[0010]

[0011] Where OS is the vessel itself, TS is other vessels, and a is the safe passage distance, a = arcsin(r S / d), where d is the distance between the ship and the target ship, V O This is the speed of the ship, V. T It is the speed of the target ship, A Z C is the azimuth angle from the position of this ship to the target ship. T It is the target ship's course. When the ship is on the collision path, C O The relative motion is as follows:

[0012]

[0013] V R and C R These are the relative speed and heading of the target ship relative to our ship, respectively. Then, we calculate the estimated approach time and the distance at the time of approach:

[0014] DCPA=d|sin(C R -A Z +π)|;

[0015]

[0016] DCPA is the distance at approach time, and TCPa is the time at approach.

[0017] Furthermore, when the collision risk area overlaps with the sensing area of ​​the grid sensor, the component with the largest overlapping area of ​​the state vector is set to 1, and the component without overlap is set to 0. If the sensing area of ​​the grid sensor overlaps with the collision risk area, the state vector closest to the ship itself is set to 1.

[0018] Furthermore, the target-driven reward and heading deviation reward operate in the target driving mode. When the collision risk area does not overlap with the sensing area of ​​the grid sensor, the ship itself maintains its heading and sails toward the target point.

[0019] Furthermore, the collision avoidance reward and rule reward operate in the collision avoidance mode. When the collision risk area overlaps with the sensing area of ​​the grid sensor, the Q value of the decision layer is updated based on the collision avoidance reward and rule reward.

[0020] According to a second aspect of the present invention, an automatic collision avoidance device for ships is provided, the device comprising:

[0021] A model generation module is used to construct a collision avoidance model based on deep reinforcement learning training to make avoidance decisions. The collision avoidance model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, which is used to detect information about the ship itself and its surrounding environment as input to the collision avoidance model. The decision layer includes a Q-Learning algorithm, which is used to determine the optimal avoidance decision based on the input of the input layer. The action layer is used to adjust the ship's speed and heading based on the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is the reward when the ship is heading toward the target point. The heading deviation reward is the reward when the ship is heading toward the target point. The collision avoidance reward is the reward when the ship avoids a collision. The rule reward is the reward when the ship turns right to avoid a collision.

[0022] The risk calculation module is used to acquire the status information of other vessels when the grid sensor detects other vessels. The status information includes speed and heading. Based on the status information, the module calculates the collision risk area.

[0023] The collision avoidance module is used to determine whether it is a collision avoidance vessel or a hold-off vessel when the collision risk area overlaps with the sensing area of ​​the grid sensor. If it is a hold-off vessel, it maintains its course and speed. If it is a collision avoidance vessel, it inputs the status information of other vessels into the collision avoidance model to calculate the collision avoidance decision. After the decision layer executes the collision avoidance decision, it updates the status information of itself and other vessels into the environmental information. This process is repeated until the distance between the vessel and other vessels is greater than a threshold or less than a threshold and there is no collision risk.

[0024] According to a third aspect of this disclosure, an automatic collision avoidance device for ships is provided, comprising: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to implement the automatic collision avoidance method for ships as described above.

[0025] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described automatic collision avoidance method for ships.

[0026] The technical solution disclosed herein has the following beneficial effects:

[0027] Simplified Input and Computation: By introducing a virtual grid sensor as input, this method quantifies the dimensionality of the observed state, avoiding the problem of excessively high state space dimensionality, reducing computational complexity, and improving the model's execution efficiency. The model's simple structure and low input dimensionality enable it to execute quickly in real-time in complex environments. In particular, by defining the risk collision occurrence region, the complexity of the state space is reduced, and the convergence speed of the reinforcement learning algorithm is improved.

[0028] By designing a simplified navigation scenario judgment method, the model can distinguish between avoiding a ship and keeping up with it, without adding extra model complexity, thus improving the model's generalization ability and practicality. Attached Figure Description

[0029] Figure 1 This is a flowchart of an automatic collision avoidance method for ships, as described in the embodiments of this specification.

[0030] Figure 2 This is a structural block diagram of an automatic collision avoidance device for ships, as described in the embodiments of this specification.

[0031] Figure 3 The device used in the embodiments of this specification to perform an automatic collision avoidance method for ships;

[0032] Figure 4 This specification describes a computer-readable storage medium that stores a method for automatic collision avoidance of ships, as exemplified in the embodiments of this specification. Detailed Implementation

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure may be practiced with one or more of the specific details omitted, or other methods, components, systems, steps, etc., may be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0034] Furthermore, the accompanying drawings are merely illustrative of this disclosure. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0035] like Figure 1 As shown in the embodiments of this specification, an automatic collision avoidance method for ships is provided, and the method is applied to a ship communication system. The method specifically includes the following steps S101 to S103:

[0036] In step S101, a collision avoidance model is constructed based on deep reinforcement learning training to formulate avoidance decisions. The collision avoidance model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, which is used to detect information about the ship itself and its surrounding environment as input to the collision avoidance model. The decision layer includes a Q-Learning algorithm, which is used to determine the optimal avoidance decision based on the input of the input layer. The action layer is used to adjust the ship's speed and heading based on the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is the reward when the ship is heading toward the target point. The heading deviation reward is the reward when the ship is heading toward the target point. The collision avoidance reward is the reward when the ship avoids a collision. The rule reward is the reward when the ship turns right to avoid a collision.

[0037] The model is structured as follows: The input layer uses a grid sensor as its input tool. This grid sensor is a virtual sensor that detects the ship's own state and information about its surrounding environment, using this information as model input. The decision layer employs a Q-Learning algorithm to process the input layer's information and determine the optimal avoidance decision. Q-Learning is a model-free reinforcement learning algorithm that learns an action-value function (Q-function) to select the best action. The action layer, based on the decision layer's avoidance decision, is responsible for adjusting the ship's speed and heading, i.e., executing the avoidance action. The reward layer provides feedback to the decision layer to update and optimize its output. Specifically, the reward layer consists of the following components:

[0038] Goal Reward: This is a reward given to ships as they sail toward a target point, encouraging them to move toward the intended goal.

[0039] Heading-Error Reward: This reward is given when a ship's heading matches the heading of the target point, encouraging the ship to maintain the correct heading.

[0040] Collision Avoidance Reward: This reward is given to ships that successfully avoid collisions with other vessels, encouraging them to take evasive action.

[0041] COLREGs Reward: According to the International Regulations for Preventing Collisions at Sea (COLREGs), a vessel will receive this reward when it takes the action of turning to starboard when it needs to avoid a collision.

[0042] During automatic collision avoidance, the model receives input from grid sensors in real time. The Q-Learning algorithm evaluates various possible avoidance strategies based on this input and selects the strategy with the highest expected reward. Then, the action layer adjusts the ship's course and speed according to this strategy to avoid a collision. Simultaneously, the reward layer provides corresponding reward signals based on the ship's behavior and the avoidance result. These signals are fed back to the decision layer to adjust and improve future avoidance decisions.

[0043] In the steps described above, the model was designed to comply with the International Regulations for Preventing Collisions at Sea (COLREGs). Specifically, during training, the model was designed to tend to turn to starboard when collision avoidance is required, which aligns with COLREGs. The model was trained in a variety of complex multi-ship encounter scenarios to improve its generalization and self-learning capabilities.

[0044] In step S102, when the grid sensor detects other vessels, it acquires the status information of the other vessels, including speed and heading, and calculates the collision risk area based on the status information.

[0045] The grid sensor, as part of the input layer, senses and acquires the status information of other nearby vessels. This status information mainly includes speed and heading. Speed ​​refers to the current speed of other vessels, which helps predict their future positions, and heading refers to their current heading, which is also crucial for predicting their navigation paths. With the status information of other vessels, the collision risk zone is calculated based on this data. The risk zone is mainly determined by: the vessel's current position and heading, the vessel's speed and expected navigation path, and the safe passage distance, which is the minimum distance that vessels should maintain to avoid a collision. By calculating the risk zone, the probability of a collision between a vessel and a target vessel at a future point in time can be assessed. This involves analyzing the relative motion of the two vessels, including their relative speed and relative heading. If the risk zone intersects with the detection range of the grid sensor, it indicates a collision risk. In this case, the severity and urgency of the collision are assessed based on the specific location of the risk zone and its relative position with other vessels. During the calculation, the calculated collision risk zone is provided as input to the decision layer, specifically the Q-Learning algorithm, to formulate appropriate avoidance decisions. These decisions can include changing course or speed to avoid entering the risk zone and reduce the likelihood of a collision. During the ship's voyage, grid sensors will continuously monitor the surrounding environment and update the status information of other vessels in real time. This means that the calculation of the collision risk zone will also be a dynamic process, constantly adjusting as the environment changes.

[0046] In step S103, when the collision risk area overlaps with the sensing area of ​​the grid sensor, it is determined whether the vessel is a avoidance vessel or a hold-off vessel. If it is a hold-off vessel, it maintains its course and speed. If it is an avoidance vessel, it inputs the status information of other vessels into the collision avoidance model to calculate the avoidance decision. After the avoidance decision is executed at the decision layer, the status information of the vessel and other vessels is updated into the environmental information. This process is repeated until the distance between the vessel and other vessels is greater than a threshold or the distance between the vessel and other vessels is less than a threshold and there is no collision risk.

[0047] When the grid sensors detect an overlap between the collision risk area and the sensing area, the system first needs to determine whether the vessel is to avoid a collision or maintain its course. This determination is based on international maritime collision avoidance regulations and other relevant factors, similar to road traffic rules, such as turning to avoid straight-going vessels. If the vessel is determined to maintain its course, it will retain its current course and speed, meaning that no avoidance action is required in the current situation. If the vessel is determined to avoid a collision, the status information of other vessels (such as speed and course) needs to be input into the collision avoidance model. This information will be used to calculate the avoidance decision. Then, the decision layer of the collision avoidance model, particularly the Q-Learning algorithm, will calculate the optimal avoidance decision based on the input status information, including changing course, adjusting speed, or other avoidance maneuvers. The action layer will then execute these decisions, adjusting the vessel's course and / or speed to avoid a collision. After the avoidance maneuver is executed, the status information of the vessel and surrounding vessels will be updated, including their positions, speeds, and headings. This updated information will be fed back into the environmental information system. Furthermore, the relative positions and status of the vessels will be continuously monitored, and the above process will be repeated. That is, the above process is a dynamic cycle until any of the following conditions are met:

[0048] The distance between this vessel and other vessels is greater than a certain preset safety threshold, and it is considered that there is no risk of collision.

[0049] The vessel was less than the threshold distance from other vessels, but avoidance maneuvers ensured there was no risk of collision.

[0050] The purpose of step S103 is to ensure that, in a multi-vehicle environment, the vessel can effectively identify collision risks and make corresponding avoidance decisions to ensure the safety of the vessel. Through this continuous risk assessment and decision-making process, the vessel's automatic avoidance system is ensured to respond promptly and effectively in the ever-changing marine environment, reducing the occurrence of maritime accidents.

[0051] In one embodiment, the collision risk area is calculated as follows:

[0052]

[0053] Where OS is the vessel itself, TS is other vessels, and a is the safe passage distance, a = arcsin(r S / d), where d is the distance between the ship and the target ship, V O This is the speed of the ship, V. T It is the speed of the target ship, A Z C is the azimuth angle from the position of this ship to the target ship. T It is the target ship's course. When the ship is on the collision path, C O The relative motion is as follows:

[0054]

[0055]

[0056] V R and C R These are the relative speed and heading of the target ship relative to our ship, respectively. Then, we calculate the estimated approach time and the distance at the time of approach:

[0057] DCPA=d|sin(C R -A Z +π)|;

[0058]

[0059] DCPA is the distance at approach time, and TCPA is the time at approach.

[0060] The calculations will be used to assess whether the two vessels are on a potential collision course and to help decision-makers determine whether evasive action is necessary. If the DCPA is less than the safe distance, or the TCPA indicates that the two vessels will approach each other in a very short time, evasive action is required. These calculations provide a quantitative risk assessment for automatic collision avoidance, enabling more accurate and timely avoidance decisions.

[0061] In one embodiment, when the collision risk area overlaps with the sensing area of ​​the grid sensor, the component with the largest overlapping area of ​​the state vector is set to 1, and the component without overlap is set to 0. If the sensing area of ​​the grid sensor overlaps with the collision risk area, the state vector closest to the ship itself is set to 1.

[0062] Specifically, when a collision risk area overlaps with the sensing area of ​​a grid sensor, a specific method is used to update the state vector for use in the collision avoidance model. For example, the state vector is a fixed-dimensional vector representing the environment surrounding the ship. When no collision risk is detected, all components of the state vector are initialized to 0. When the grid sensor detects a collision risk area, it checks whether the collision risk area overlaps with any of the sensor's sensing areas (grid cells). If the collision risk area overlaps with a grid cell, and this grid cell has the largest overlap area among all overlapping grid cells, then the state vector component corresponding to this grid cell will be set to 1, indicating that this grid cell has detected the maximum collision risk. If the collision risk area does not overlap with any grid cell, the state vector remains 0, indicating that no collision risk has been detected. When the collision risk area overlaps with multiple grid cells, the grid cell closest to the ship is selected. The state vector component corresponding to this grid cell is set to 1, while the components corresponding to other overlapping grid cells remain 0 or are not selected. The update of the state vector reflects the current collision risk situation in the environment. This updated state vector is then used as input to the collision avoidance model, specifically to the Q-Learning algorithm in the decision layer, to compute avoidance decisions. During navigation, the grid sensors continuously monitor the surrounding environment and repeat the above detection and update process. In this way, the collision avoidance model can accurately identify and assess collision risks around the ship and make corresponding avoidance decisions. The state vector setting ensures that the model can focus on the most pressing collision risks, improving the effectiveness and real-time performance of avoidance decisions.

[0063] In one embodiment, the target-driven reward and heading deviation reward operate in target driving mode. When the collision risk area does not overlap with the sensing area of ​​the grid sensor, the ship itself maintains its heading and sails toward the target point.

[0064] In target-driven mode, the ship's primary task is to navigate along a predetermined route towards the target point. In this mode, target-driven rewards and heading deviation rewards work together to ensure the ship maintains the correct course. Specifically, target-driven rewards encourage the ship to move towards the target point. As the ship moves along the route towards the target point, it receives a positive reward, which reinforces the current navigation behavior. The reward value is generally inversely proportional to the distance from the target point; the closer to the target point, the greater the reward. Heading deviation rewards are used to reduce the deviation between the ship's current course and the target course. If the ship's course aligns with the target course, it receives a positive reward. This reward value is inversely proportional to the angle of deviation between the ship's current course and the target course; the smaller the heading deviation, the greater the reward. Target-driven mode refers to a situation where the collision risk area does not overlap with the sensing area of ​​the grid sensors, meaning no collision risk is detected, and the ship can safely continue towards the target point. In this mode, the ship maintains its current course and speed, while target-driven rewards and heading deviation rewards work together to ensure the ship navigates along the most direct and efficient path to the target point. In target driving mode, the vessel also needs to continuously monitor the surrounding environment so that it can quickly switch to avoidance mode when a new collision risk is detected. If the area of ​​collision risk overlaps with the sensing area of ​​the grid sensor, the vessel will switch from target driving mode to avoidance mode, i.e., execute steps S101-103 as described above. Through this embodiment, the vessel can effectively navigate to the target point while ensuring safety. Target driving reward and heading deviation reward play a key role in target driving mode, ensuring that the vessel maintains the correct course and heading.

[0065] In one embodiment, the collision avoidance reward and rule reward operate in the collision avoidance mode. When the collision risk area overlaps with the sensing area of ​​the grid sensor, the Q value of the decision layer is updated based on the collision avoidance reward and rule reward.

[0066] When the area of ​​potential collision overlaps with the sensing area of ​​the grid sensors, indicating a possible collision risk, the ship's automatic collision avoidance system switches to collision avoidance mode. In this mode, collision avoidance rewards and rule rewards are used to update the Q-value in the decision-making layer, thereby guiding avoidance decisions. Specifically, collision avoidance rewards are activated when the ship takes avoidance actions to reduce the risk of collision. If the avoidance action is effective, the ship receives a positive reward, which can reinforce avoidance behavior. The magnitude of the reward may be related to the effectiveness, timeliness, or relative position of the avoidance action with other ships. Rule rewards are used to encourage ships to comply with international maritime collision avoidance regulations. For example, according to the international maritime collision avoidance regulations, ships should turn appropriately to starboard to avoid a collision when encountering each other. If the ship's avoidance decision and actions comply with the requirements of the international maritime collision avoidance regulations, it will receive an additional positive reward. The Q-value, or action value function, is updated using the Q-Learning algorithm and represents the expected reward for taking a specific action in a given state. In collision avoidance mode, when the ship takes an avoidance action, the Q-value is updated based on the collision avoidance reward and rule rewards, thus influencing future avoidance decisions. The decision-making layer selects the optimal avoidance maneuver based on the updated Q-value. This includes changing course, decelerating, or other avoidance strategies. The action layer then executes these decisions, adjusting the ship's speed and course to avoid a collision with the target vessel. In avoidance mode, the ship's status and surrounding environment are continuously monitored to ensure the effectiveness of the avoidance maneuver and adjustments are made as needed. If the collision risk area no longer overlaps with the sensing area of ​​the grid sensors, or if the collision risk decreases to an acceptable level, the system switches back to target driving mode. This implementation allows for automatic adjustment of behavior through reinforcement learning when a collision risk is detected, taking appropriate avoidance measures while ensuring compliance with maritime collision avoidance rules, thus improving navigational safety.

[0067] Based on the same line of thought, such as Figure 2 As shown in the embodiments of this specification, an automatic collision avoidance device for ships is also provided, the device comprising:

[0068] The model generation module 201 is used to construct a collision avoidance model based on deep reinforcement learning training to formulate avoidance decisions. The collision avoidance model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, which is used to detect information about the ship itself and its surrounding environment as input to the collision avoidance model. The decision layer includes a Q-Learning algorithm, which is used to determine the optimal avoidance decision based on the input of the input layer. The action layer is used to adjust the ship's speed and heading based on the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is the reward when the ship is heading toward the target point. The heading deviation reward is the reward when the ship is heading toward the target point. The collision avoidance reward is the reward when the ship avoids a collision. The rule reward is the reward when the ship turns right to avoid a collision.

[0069] Risk calculation module 202 is used to acquire the status information of other ships when the grid sensor detects other ships, the status information including speed and heading, and calculate the collision risk area based on the status information;

[0070] The avoidance module 203 is used to determine whether it is an avoidance vessel or a hold-off vessel when the collision risk area overlaps with the sensing area of ​​the grid sensor. If it is a hold-off vessel, it maintains its course and speed unchanged. If it is an avoidance vessel, it inputs the status information of other vessels into the collision avoidance model to calculate the avoidance decision. After the avoidance decision is executed at the decision layer, it updates the status information of itself and other vessels into the environmental information. This process is repeated until the distance between the vessel and other vessels is greater than a threshold or less than a threshold and there is no collision risk.

[0071] The advantages of the device are as follows:

[0072] Simplified Input and Computation: By introducing a virtual grid sensor as input, this method quantifies the dimensionality of the observed state, avoiding the problem of excessively high state space dimensionality, reducing computational complexity, and improving model execution efficiency. The simple model structure and low input dimensionality enable it to perform fast real-time execution in complex environments. In particular, by defining the risk collision occurrence region, the complexity of the state space is reduced, and the convergence speed of the reinforcement learning algorithm is improved. Through the design of a simplified navigation scenario judgment method, the model can distinguish between avoiding other ships and keeping pace with them without adding additional model complexity, thus improving the model's generalization ability and practicality.

[0073] Based on the same idea, this specification also provides an automatic collision avoidance device for ships, such as... Figure 3 As shown.

[0074] The ship automatic collision avoidance device can be the terminal device or server provided in the above embodiments.

[0075] Automatic collision avoidance systems for ships can vary considerably depending on their configuration and performance. They may include one or more processors 301 and memory 302, with memory 302 storing one or more application programs or data. Memory 302 may include readable media in the form of volatile memory cells, such as random access memory (RAM) and / or cache memory, and may further include read-only memory cells. The application programs stored in memory 302 may include one or more program modules (not shown in the figures), including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Furthermore, processor 301 may be configured to communicate with memory 302 to execute a series of computer-executable instructions stored in memory 302 on the automatic collision avoidance system. The ship's automatic collision avoidance system may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more I / O interfaces (input / output interfaces) 305, one or more external devices 306 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that enable users to interact with the device, and / or with any device that enables the device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed through I / O interfaces 305. Furthermore, the device can also communicate with one or more networks (e.g., local area networks (LANs)) through wired or wireless interfaces 304.

[0076] Specifically, in this embodiment, the ship's automatic collision avoidance device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the ship's automatic collision avoidance device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0077] A collision avoidance model is constructed based on deep reinforcement learning to make avoidance decisions. The collision avoidance model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor used to detect information about the ship itself and its surrounding environment as input to the collision avoidance model. The decision layer includes a Q-Learning algorithm used to determine the optimal avoidance decision based on the input from the input layer. The action layer is used to adjust the ship's speed and heading based on the avoidance decision. The reward layer updates the output of the decision layer and includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is the reward for the ship heading toward the target point. The heading deviation reward is the reward for the ship heading toward the target point. The collision avoidance reward is the reward for the ship avoiding a collision. The rule reward is the reward for the ship turning right to avoid a collision.

[0078] When the grid sensor detects other vessels, it acquires the status information of the other vessels, including speed and heading, and calculates the collision risk area based on the status information.

[0079] When the collision risk area overlaps with the sensing area of ​​the grid sensor, it determines whether it is a avoidance vessel or a hold-off vessel. If it is a hold-off vessel, it maintains its course and speed. If it is an avoidance vessel, it inputs the status information of other vessels into the collision avoidance model to calculate the avoidance decision. After the avoidance decision is executed at the decision layer, the status information of itself and other vessels is updated into the environmental information. This process is repeated until the distance between the vessel and other vessels is greater than a threshold or less than a threshold and there is no collision risk.

[0080] Based on the same idea, exemplary embodiments of this disclosure also provide a computer-readable storage medium storing a program product capable of implementing the methods described above. In some possible embodiments, various aspects of this disclosure can also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Automatic Collision Avoidance Method for Ships" section of this specification according to various exemplary embodiments of this disclosure.

[0081] refer to Figure 4As shown, a program product 400 for implementing the above-described method according to an exemplary embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, system, or device.

[0082] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0083] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, system, or device.

[0084] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0085] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0086] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal system, or network device, etc.) to execute the method according to the exemplary embodiments of this disclosure.

[0087] Furthermore, the above figures are merely illustrative representations of the processes included in the methods according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0088] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0089] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A method for automatic collision avoidance of a ship, characterized in that, The method comprises: The method comprises: When the grid sensor senses other ships, state information of the other ships is obtained, the state information comprising speed and heading, and a collision risk occurrence area is calculated based on the state information; when the collision risk occurrence area is calculated, the following is calculated: ； where OS is the own ship, TS is the other ship, is the safe passing distance, , is the distance between the own ship and the target ship, is the speed of the own ship, is the speed of the target ship, is the bearing from the own ship to the target ship position, is the heading of the target ship, when the ships are on a collision course, the relative motion is as follows: ； ； and are the relative speed and heading of the target ship with respect to the own ship, respectively, and the time to go and the distance at go-aside are calculated. ； ； is the distance at the time of approach, is the approach time; When the collision risk occurrence area overlaps with the sensing area of the grid sensor, it is determined whether the ship is an avoidance ship or a keep ship; if the ship is a keep ship, the heading and speed are kept unchanged; if the ship is an avoidance ship, the state information of the other ships is input into the anti-collision model to calculate the avoidance decision; after the avoidance decision is executed in the decision layer, the state information of the ship and the other ships is updated into the environmental information, and the process is repeated until the distance between the ship and the other ships is greater than a threshold or the distance between the ship and the other ships is less than a threshold and there is no collision risk.

2. The method of claim 1, wherein, When the collision risk occurrence area overlaps with the sensing area of the grid sensor, the component with the largest state vector overlap area is set to 1, and the component without overlap is set to 0; if the sensing area of the grid sensor overlaps with the collision risk occurrence area, the state vector closest to the ship is set to 1.

3. The method of claim 1, wherein, The target driving reward and the heading deviation reward work in a target driving mode; when the collision risk occurrence area does not overlap with the sensing area of the grid sensor, the ship keeps the heading and sails to the target point.

4. The method of claim 1, wherein, The collision avoidance reward and the rule reward work in a collision avoidance mode; when the collision risk occurrence area overlaps with the sensing area of the grid sensor, the Q value of the decision layer is updated based on the collision avoidance reward and the rule reward.

5. An automatic collision avoidance device for a ship, characterized by The device comprises: The model generation module is configured to construct an anti-collision model based on deep reinforcement learning training to make an avoidance decision, the anti-collision model comprising an input layer, a decision layer, an action layer, and a reward layer, the input layer comprising a grid sensor configured to detect environmental information of the ship itself and surroundings as input of the anti-collision model, the decision layer comprising a Q-Learning algorithm configured to determine an optimal avoidance decision according to the input of the input layer, the action layer configured to adjust a speed and a heading of the ship according to the avoidance decision, and the reward layer configured to update output of the decision layer, which comprises a target driving reward, a heading deviation reward, a collision avoidance reward, and a rule reward, the target driving reward being a reward when the ship sails towards a target point, the heading deviation reward being a reward when the heading of the ship sails towards the target point, the collision avoidance reward being a reward when the ship avoids collision, and the rule reward being a reward when the ship adopts a right turn to avoid collision. The risk calculation module is configured to acquire state information of other ships when the grid sensor senses the other ships, the state information comprising speed and heading, and calculate a collision risk occurrence area based on the state information. ； where OS is the own ship, TS is the other ship, is the safe passing distance, , is the distance between the own ship and the target ship, is the speed of the own ship, is the speed of the target ship, is the azimuth angle from the own ship to the target ship position, is the heading of the target ship, when the ships are on a collision course, the relative motion is as follows: ； ； and are the relative speed and heading of the target ship with respect to the own ship, respectively, and the time to go and the distance at go-aside are calculated. ； ； is the distance at the time of approach, is the approach time; The avoidance module is configured to determine whether the ship itself is an avoidance ship or a keep ship when the collision risk occurrence area overlaps with a sensing area of the grid sensor, keep the heading and speed unchanged if the ship itself is the keep ship, input the state information of the other ships into the anti-collision model to calculate the avoidance decision if the ship itself is the avoidance ship, update the state information of the ship itself and the other ships into the environmental information after the avoidance decision is executed in the decision layer, and repeat the execution until the distance between the ship itself and the other ships is greater than a threshold or the distance between the ship itself and the other ships is less than the threshold and there is no collision risk.

6. An automatic collision avoidance equipment for a ship, characterized in that, The system comprises: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to: construct an anti-collision model based on deep reinforcement learning training to make an avoidance decision, the anti-collision model comprising an input layer, a decision layer, an action layer, and a reward layer, the input layer comprising a grid sensor configured to detect environmental information of the ship itself and surroundings as input of the anti-collision model, the decision layer comprising a Q-Learning algorithm configured to determine an optimal avoidance decision according to the input of the input layer, the action layer configured to adjust a speed and a heading of the ship according to the avoidance decision, and the reward layer configured to update output of the decision layer, which comprises a target driving reward, a heading deviation reward, a collision avoidance reward, and a rule reward, the target driving reward being a reward when the ship sails towards a target point, the heading deviation reward being a reward when the heading of the ship sails towards the target point, the collision avoidance reward being a reward when the ship avoids collision, and the rule reward being a reward when the ship adopts a right turn to avoid collision. When the grid sensor senses other ships, state information of the other ships is acquired, the state information including speed and heading, and a collision risk occurrence area is calculated based on the state information; when the collision risk occurrence area is calculated, the following is calculated: ； where OS is the own ship, TS is the other ship, is the safe passing distance, , is the distance between the own ship and the target ship, is the speed of the own ship, is the speed of the target ship, is the bearing from the own ship to the target ship position, is the heading of the target ship, when the ships are on a collision course, the relative motion is as follows: ； ； and are the relative speed and heading of the target ship with respect to the own ship, respectively, and the time to go and the distance at go-aside are calculated. ； ； is the distance at the time of approach, is the approach time; When the collision risk occurrence area overlaps with a sensing area of the grid sensor, it is determined whether the ship is a keep ship or an avoidance ship, if the ship is a keep ship, the heading and speed are kept unchanged, if the ship is an avoidance ship, the state information of the other ships is input into the anti-collision model to calculate the avoidance decision, after the avoidance decision is executed in the decision layer, the state information of the ship and the other ships is updated into the environment information, and the execution is repeated until the distance between the ship and the other ships is greater than a threshold or the distance between the ship and the other ships is less than the threshold and there is no collision risk.

7. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by a processor to implement the ship automatic avoidance anti-collision method according to any one of claims 1 to 4.

Images