Ship automatic avoidance anti-collision method, device and equipment and storage medium

Abstract

The invention belongs to the field of ship automation control, and provides a ship automatic avoidance anti-collision method, device and equipment and a storage medium. The method comprises the following steps: constructing an anti-collision model based on deep reinforcement learning training to make an avoidance decision; when the grid sensor senses other ships, state information of the other ships is obtained, and a collision risk occurrence area is calculated based on the state information; when the collision risk occurrence area is overlapped with the sensing area of the grid sensor, judging whether the ship itself is an avoidance ship or a holding ship, if the ship itself is the holding ship, keeping the course and the speed unchanged, if the ship itself is the avoidance ship, calculating an avoidance decision, after the avoidance decision is executed, updating state information of the ship itself and other ships into environment information, and repeating the execution; the distance between the ship and other ships is larger than the threshold value or smaller than the threshold value, and there is no collision risk; the method aims at solving the problems that a traditional ship collision avoidance technology is not suitable for complex sea conditions and is high in calculation complexity.

Description

Technical Field

[0001] The present invention relates to the field of ship automation control, and in particular to a method, device, equipment and storage medium for automatic ship avoidance and anti-collision. Background Art

[0002] With the continuous increase in global maritime traffic, maritime traffic is becoming increasingly congested. According to statistics, 89% to 95% of maritime accidents are caused by human factors. This highlights that in a complex navigation environment, crew members are prone to making mistakes in their decision-making process. Therefore, reducing maritime accidents, especially realizing ship collision avoidance operations through automated means, has become a key research direction in the field of marine engineering. Traditional model-based collision avoidance methods, such as artificial potential field method, speed barrier method, A* algorithm and particle swarm optimization algorithm, although they perform well in specific scenarios, have limitations in dealing with complex and changeable actual sea conditions. These methods not only have complex models and high computational overhead, but also lack self-learning capabilities and are difficult to adapt to dynamically changing environments. Summary of the invention

[0003] In view of the above technical problems, the present invention provides a method, device, equipment and storage medium for automatic ship collision avoidance, aiming to solve the problem that traditional ship collision avoidance technology is difficult to adapt to complex sea conditions and has high computational complexity.

[0004] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0005] According to one aspect of the present invention, a method for automatic ship avoidance and collision prevention is disclosed, the method comprising:

[0006] An anti-collision model is constructed based on deep reinforcement learning training to make avoidance decisions. The anti-collision model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, which is used to detect environmental information of the ship itself and its surroundings as input to the anti-collision model. The decision layer includes a Q-Learning algorithm, which is used to determine the optimal avoidance decision according to the input of the input layer. The action layer is used to adjust the speed and heading of the ship according to the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is a reward for the ship when it is sailing towards the target point, the heading deviation reward is a reward for the ship when its heading is sailing towards the target point, the collision avoidance reward is a reward for the ship when it avoids a collision, and the rule reward is a reward for the ship when it turns right to avoid a collision.

[0007] When the grid sensor senses other ships, status information of the other ships is obtained, the status information including speed and heading, and a collision risk occurrence area is calculated based on the status information;

[0008] When the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor, it is determined whether the ship itself is an avoiding ship or a holding ship. If it is a holding ship, the heading and speed are kept unchanged. If it is an avoiding ship, the status information of other ships is input into the anti-collision model to calculate the avoidance decision. After the decision layer executes the avoidance decision, the status information of itself and other ships is updated in the environmental information, and the execution is repeated until the distance between the ship itself and other ships is greater than a threshold or the distance between the ship itself and other ships is less than a threshold and there is no collision risk.

[0009] Furthermore, when calculating the collision risk occurrence area, the calculation is as follows:

[0010]

[0011] Where OS is the ship itself, TS is other ships, a is the safe passing distance, a = arcsin (r S / d), d is the distance between own ship and target ship, V O is the speed of own ship, V T is the speed of the target ship, A Z is the azimuth from own ship to target ship, C T is the heading of the target ship. When the ship is on the collision course, C O The relative motion is as follows:

[0012]

[0013] V R and C R The relative speed and course of the target ship relative to the own ship are calculated, and then the estimated approach time and distance between the two ships are calculated:

[0014] DCPA=d|sin(C R -A Z +π)|;

[0015]

[0016] DCPA is the distance at the time of approach, and TCPa is the time of approach.

[0017] Furthermore, when the collision risk area overlaps with the sensing area of ​​the grid sensor, the component of the state vector with the largest overlapping area is set to 1, and the setting without overlap is set to 0. If the sensing area of ​​the grid sensor overlaps with the collision risk area, the state vector closest to the ship itself is set to 1.

[0018] Furthermore, the target driving reward and the heading deviation reward work in the target driving mode. When the collision risk occurrence area does not overlap with the sensing area of ​​the grid sensor, the ship itself maintains its course and sails towards the target point.

[0019] Furthermore, the collision avoidance reward and rule reward work in a collision avoidance mode. When the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor, the Q value of the decision layer is updated based on the collision avoidance reward and rule reward.

[0020] According to a second aspect of the present invention, there is provided a device for automatic ship avoidance and collision prevention, the device comprising:

[0021] A model generation module, used to build an anti-collision model based on deep reinforcement learning training to make avoidance decisions, the anti-collision model includes an input layer, a decision layer, an action layer, and a reward layer, the input layer includes a grid sensor, the grid sensor is used to detect the environmental information of the ship itself and the surroundings as the input of the anti-collision model, the decision layer includes a Q-Learning algorithm, the Q-Learning algorithm is used to determine the optimal avoidance decision according to the input of the input layer, the action layer is used to adjust the speed and heading of the ship according to the avoidance decision, and the reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward, the target-driven reward is a reward for the ship when it sails toward the target point, the heading deviation reward is a reward for the ship when its heading is sailing toward the target point, the collision avoidance reward is a reward for the ship when it avoids a collision, and the rule reward is a reward for the ship when it turns right to avoid a collision;

[0022] A risk calculation module, used for obtaining status information of other ships when the grid sensor senses other ships, the status information including speed and heading, and calculating a collision risk occurrence area based on the status information;

[0023] The avoidance module is used to determine whether the ship itself is an avoidance ship or a holding ship when the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor. If it is a holding ship, the heading and speed are kept unchanged. If it is an avoidance ship, the status information of other ships is input into the anti-collision model to calculate the avoidance decision. After the decision layer executes the avoidance decision, the status information of the ship itself and other ships is updated in the environmental information, and the execution is repeated until the distance between the ship itself and other ships is greater than a threshold or the distance between the ship itself and other ships is less than a threshold and there is no collision risk.

[0024] According to a third aspect of the present disclosure, there is provided a ship automatic avoidance and collision prevention device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor implements the ship automatic avoidance and collision prevention method as described above;

[0025] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the automatic ship avoidance and collision prevention method as described above is implemented.

[0026] The technical solution disclosed in this disclosure has the following beneficial effects:

[0027] Simplify input and calculation: Introduce virtual grid sensors as input. This method quantifies the dimension of the observed state, avoids the problem of too high a dimension of the state space, reduces computational complexity, and improves the execution efficiency of the model. The model has a simple structure and low input dimension, which enables it to have fast real-time execution capabilities in complex environments. In particular, by setting the risk collision area, the complexity of the state space is reduced and the convergence speed of the reinforcement learning algorithm is improved.

[0028] By designing a simplified navigation situation judgment method, the model can distinguish between avoiding ships and keeping ships without adding additional model complexity, which improves the generalization ability and practicality of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a method for automatic ship avoidance and collision prevention in an embodiment of this specification;

[0030] Figure 2 This is a structural block diagram of a ship automatic avoidance and anti-collision device in an embodiment of this specification;

[0031] Figure 3 A device for executing a method for automatic ship avoidance and collision prevention in an embodiment of this specification;

[0032] Figure 4 It is a computer-readable storage medium storing a method for automatic ship avoidance and collision prevention in an embodiment of this specification. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, systems, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor systems and / or microcontroller systems.

[0035] like Figure 1 As shown, the embodiment of this specification provides a method for automatic ship avoidance and collision prevention, which is applied to a ship communication system. The method may specifically include the following steps S101 to S103:

[0036] In step S101, an anti-collision model is constructed based on deep reinforcement learning training to make an avoidance decision. The anti-collision model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, and the grid sensor is used to detect environmental information of the ship itself and its surroundings as an input of the anti-collision model. The decision layer includes a Q-Learning algorithm, and the Q-Learning algorithm is used to determine the optimal avoidance decision according to the input of the input layer. The action layer is used to adjust the speed and heading of the ship according to the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is a reward for the ship when it is sailing towards the target point, the heading deviation reward is a reward for the ship when its heading is sailing towards the target point, the collision avoidance reward is a reward for the ship when it avoids a collision, and the rule reward is a reward for the ship when it turns right to avoid a collision.

[0037] Among them, the input layer uses grid sensors as its input tool. The grid sensor is a virtual sensor that can detect the state of the ship itself and the information of the surrounding environment, and use this information as the input of the model. The Q-Learning algorithm of the decision layer is used to process the information of the input layer and determine the optimal avoidance decision. Q-Learning is a model-free reinforcement learning algorithm that selects the best action by learning an action value function (Q-function). The action layer is responsible for adjusting the speed and heading of the ship according to the avoidance decision of the decision layer, that is, performing avoidance actions. The role of the reward layer is to provide feedback to the decision layer to update and optimize its output. Specifically, the reward layer consists of the following parts:

[0038] Goal Reward: This reward is obtained when the ship is sailing towards the target point, which encourages the ship to move towards the predetermined goal.

[0039] Heading-Error Reward: When the ship's heading is consistent with the heading of the target point, this reward will be obtained to encourage the ship to maintain the correct heading.

[0040] Collision Avoidance Reward: This reward is given when a ship successfully avoids a collision with another ship, encouraging the ship to take avoidance action.

[0041] COLREGs Reward: This reward is awarded when a vessel takes the action of turning right primarily when an avoidance action is required under the International Regulations for Preventing Collisions at Sea (COLREGs).

[0042] During the ship's automatic avoidance process, the model receives input from grid sensors in real time. The Q-Learning algorithm evaluates various possible avoidance strategies based on these inputs and selects the strategy with the highest expected return. The action layer then adjusts the ship's course and speed based on this strategy to avoid collisions. At the same time, the reward layer provides corresponding reward signals based on the ship's behavior and avoidance results. These signals are fed back to the decision layer to adjust and improve future avoidance decisions.

[0043] In the above steps, the model is designed to comply with the International Regulations for Preventing Collisions at Sea (COLREGs), that is, during the training process, the model is designed to tend to take actions to turn to starboard when avoidance is required, which is consistent with the provisions of COLREGs. The above model is trained in a variety of complex multi-ship encounter scenarios to improve its generalization and self-learning capabilities.

[0044] In step S102, when the grid sensor senses other ships, the status information of the other ships is acquired, the status information including speed and heading, and the collision risk occurrence area is calculated based on the status information.

[0045] Among them, the grid sensor, as part of the input layer, can sense and obtain the status information of other nearby ships. This status information mainly includes speed and heading. Speed ​​refers to the current speed of other ships, which helps to predict their future positions. Heading refers to the current heading of other ships, which is also crucial for predicting their navigation paths. With the status information of other ships, the collision risk area will be calculated based on these data. The risk area is mainly determined by: the current position and heading of the ship, the speed of the ship and the expected navigation path, and the safe passing distance. The safe passing distance refers to the minimum distance that should be maintained between ships to avoid collision. By calculating the risk area, the possibility of a collision between the ship and the target ship at a certain point in the future can be evaluated, which specifically involves analyzing the relative motion of the two ships, including their relative speed and relative heading. If the risk area intersects with the detection range of the grid sensor, this indicates that there is a collision risk. At this time, the severity and urgency of the collision will be evaluated based on the specific location of the risk area and the relative position with other ships. During the calculation, the calculated collision risk area will be provided as input to the decision layer, especially the Q-Learning algorithm, so as to make corresponding avoidance decisions. These decisions can include changing course or speed to avoid entering the risk area and reduce the possibility of collision. During the ship's navigation, the grid sensors will continuously monitor the surrounding environment and update the status information of other ships in real time. This means that the calculation of the collision risk area will also be a dynamic process, constantly adjusting as the environment changes.

[0046] In step S103, when the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor, it is determined whether the ship itself is an avoiding ship or a holding ship. If it is a holding ship, the heading and speed are kept unchanged. If it is an avoiding ship, the status information of other ships is input into the anti-collision model to calculate the avoidance decision. After the decision layer executes the avoidance decision, the status information of itself and other ships is updated in the environmental information, and the execution is repeated until the distance between the ship itself and other ships is greater than the threshold or the distance between the ship itself and other ships is less than the threshold and there is no collision risk.

[0047] Among them, when the grid sensor detects that the collision risk area overlaps with the sensing area, the system first needs to determine whether the ship is an avoidance ship or a holding ship, which is determined based on the International Regulations for Preventing Collisions at Sea and other relevant factors, that is, similar to road traffic rules, turning to avoid straight driving, etc. If the ship is judged to be a holding ship, the current course and speed will be kept unchanged, which means that in the current situation, the ship does not need to take avoidance action. If the ship is judged to be an avoidance ship, the state information of other ships (such as speed and course) needs to be input into the collision avoidance model, and this information will be used to calculate the avoidance decision. Then, the decision layer of the collision avoidance model, especially the Q-Learning algorithm, will calculate the optimal avoidance decision based on the input state information, including changing the course, adjusting the speed or other avoidance actions, and the action layer will execute these decisions and adjust the course and / or speed of the ship to avoid collision. After the avoidance action is executed, the status information of the own ship and the surrounding ships, including their positions, speeds and headings, will be updated, and the updated information will be fed back into the environmental information. In addition, the relative positions and states of the ships will be continuously monitored and the above process will be repeated. That is, the above process is a dynamic cycle until any of the following conditions is met:

[0048] The distance between the own ship and other ships is greater than a certain preset safety threshold, and it is considered that there is no risk of collision;

[0049] The distance between the own ship and other ships is less than the threshold, but the avoidance action ensures that there is no risk of collision.

[0050] The purpose of step S103 is to ensure that in a multi-ship environment, the ship can effectively identify the risk of collision and make corresponding avoidance decisions to ensure the safety of the ship. Through this continuous risk assessment and decision-making process, it is ensured that the ship automatic avoidance system can make timely and effective responses in the ever-changing marine environment, reducing the occurrence of marine traffic accidents.

[0051] In one implementation, when calculating the collision risk occurrence area, the calculation is as follows:

[0052]

[0053] Where OS is the ship itself, TS is other ships, a is the safe passing distance, a = arcsin (r S / d), d is the distance between own ship and target ship, V O is the speed of own ship, V T is the speed of the target ship, A Z is the azimuth from own ship to target ship, C T is the heading of the target ship. When the ship is on the collision course, C O The relative motion is as follows:

[0054]

[0055]

[0056] V R and C R The relative speed and course of the target ship relative to the own ship are calculated, and then the estimated approach time and distance between the two ships are calculated:

[0057] DCPA=d|sin(C R -A Z +π)|;

[0058]

[0059] DCPA is the distance at the time of approach, and TCPA is the time of approach.

[0060] The above calculation results will be used to assess whether the two ships are on a potential collision course and help decision-makers determine whether avoidance action is needed. If the DCPA is less than the safe distance, or the TCPA indicates that the two ships will approach this point in a short time, avoidance measures are required. These calculations provide quantitative risk assessments for automatic ship avoidance, enabling more accurate and timely avoidance decisions to be made.

[0061] In one embodiment, when the collision risk area overlaps with the sensing area of ​​the grid sensor, the component of the state vector with the largest overlapping area is set to 1, and the setting without overlap is set to 0. If the sensing area of ​​the grid sensor overlaps with the collision risk area, the state vector closest to the ship itself is set to 1.

[0062] Among them, when the collision risk area overlaps with the sensing area of ​​the grid sensor, a specific method will be used to update the state vector for use in the collision avoidance model. For example, the state vector is a fixed-dimensional vector used to represent the environment around the ship. When no collision risk is detected, all components of the state vector are initialized to 0. When the grid sensor senses the collision risk area, it will check whether the collision risk area overlaps with each sensing area (grid cell) of the grid sensor. If the collision risk area overlaps with a grid cell, and the overlapping area of ​​this grid cell is the largest among all overlapping grid cells, then the state vector component corresponding to this grid cell will be set to 1, which means that the grid cell has detected the largest collision risk. If the collision risk area does not overlap with any grid cell, the state vector remains 0, indicating that no collision risk is currently detected. When the collision risk area overlaps with multiple grid cells, the grid cell closest to the ship itself is selected. The state vector component corresponding to this grid cell is set to 1, while the components corresponding to other overlapping grid cells remain 0 or are not selected. The update of the state vector reflects the collision risk situation in the current environment. This updated state vector is then used as input to the collision avoidance model, specifically the Q-Learning algorithm in the decision layer, to calculate avoidance decisions. During the ship's voyage, the grid sensors continuously monitor the surrounding environment and repeat the above detection and update process. In this way, the collision avoidance model can accurately identify and assess the collision risk around the ship and make corresponding avoidance decisions accordingly. The setting of the state vector ensures that the model can focus on the most pressing collision risks, improving the effectiveness and real-time performance of avoidance decisions.

[0063] In one embodiment, the target driving reward and the heading deviation reward operate in the target driving mode, and when the collision risk occurrence area does not overlap with the sensing area of ​​the grid sensor, the ship itself maintains its course and sails toward the target point.

[0064] Among them, in the target driving mode, the main task of the ship itself is to sail to the target point along the predetermined route. In this case, the target-driven reward and the heading deviation reward play a role to ensure that the ship maintains the correct route. Specifically, the target-driven reward encourages the ship to sail towards the target point. When the ship moves along the route to the target point, it will receive a positive reward, which helps to strengthen the current navigation behavior, and the size of the reward is usually inversely proportional to the distance from the ship to the target point, that is, the closer the distance to the target point, the greater the reward value. The heading deviation reward is used to reduce the deviation between the current heading of the ship and the heading of the target point. If the heading of the ship is consistent with the heading of the target point, it will receive a positive reward. This reward value is inversely proportional to the deviation angle between the current heading of the ship and the target heading, that is, the smaller the heading deviation, the greater the reward value. The target driving mode refers to when the collision risk area does not overlap with the sensing area of ​​the grid sensor, which means that no collision risk is detected, and the ship itself can safely continue to sail to the target point. In this case, the ship maintains the current heading and speed, and the target-driven reward and heading deviation reward work together to ensure that the ship sails to the target point along the most direct and efficient path. In the target driving mode, the ship also needs to continuously monitor the surrounding environment so that it can quickly switch to the avoidance mode when a new collision risk is detected. If the collision risk area overlaps with the sensing area of ​​the grid sensor, the ship will switch from the target driving mode to the avoidance mode, that is, execute the above steps S101-103. Through this embodiment, the ship can effectively navigate to the target point while ensuring safety. The target drive reward and the heading deviation reward play a key role in the target driving mode to ensure that the ship maintains the correct route and heading.

[0065] In one embodiment, the collision avoidance reward and rule reward operate in a collision avoidance mode, and when the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor, the Q value of the decision layer is updated based on the collision avoidance reward and rule reward.

[0066] Among them, when the collision risk area overlaps with the sensing area of ​​the grid sensor, that is, when a possible collision risk is detected, the ship's automatic avoidance system will switch to the collision avoidance mode. In this mode, the collision avoidance reward and the rule reward will be used to update the Q value in the decision layer to guide the avoidance decision. Specifically, the collision avoidance reward is activated when the ship takes an avoidance action to reduce the risk of collision. If the avoidance action is effective, the ship will receive a positive reward, which can strengthen the avoidance behavior. The size of the reward may be related to the effectiveness, timeliness or relative position of the avoidance action with other ships. The rule reward is used to encourage ships to comply with the international regulations for preventing collisions at sea. For example, according to the international regulations for preventing collisions at sea, ships should turn to the starboard appropriately when meeting to avoid collisions. If the ship's avoidance decision and action meet the requirements of the international regulations for preventing collisions at sea, it will receive additional positive rewards. The Q value, or action value function, is updated by the Q-Learning algorithm and represents the expected return of taking a specific action in a given state. In the collision avoidance mode, when the ship takes an avoidance action, the Q value is updated according to the collision avoidance reward and the rule reward, thereby affecting future avoidance decisions. The decision layer selects the best avoidance action based on the updated Q value, which includes changing course, slowing down, or other avoidance strategies. The action layer then executes these decisions and adjusts the speed and course of the ship to avoid a collision with the target ship. In avoidance mode, the state of the ship and the surrounding environment are continuously monitored to ensure the effectiveness of the avoidance action and make adjustments as needed. If the area where the collision risk occurs no longer overlaps with the sensing area of ​​the grid sensor, or the collision risk is reduced to an acceptable level, switch back to the target driving mode. Through this implementation, it is possible to automatically adjust its behavior through reinforcement learning when a collision risk is detected, take appropriate avoidance measures, and ensure compliance with maritime collision avoidance regulations to improve the safety of navigation.

[0067] Based on the same idea, Figure 2 As shown, the embodiment of this specification also provides a ship automatic avoidance and anti-collision device, the device comprising:

[0068] The model generation module 201 is used to build an anti-collision model based on deep reinforcement learning training to make avoidance decisions. The anti-collision model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, and the grid sensor is used to detect the environmental information of the ship itself and the surroundings as the input of the anti-collision model. The decision layer includes a Q-Learning algorithm, and the Q-Learning algorithm is used to determine the optimal avoidance decision according to the input of the input layer. The action layer is used to adjust the speed and heading of the ship according to the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is a reward for the ship when it is sailing towards the target point, the heading deviation reward is a reward for the ship when its heading is sailing towards the target point, the collision avoidance reward is a reward for the ship when it avoids a collision, and the rule reward is a reward for the ship when it turns right to avoid a collision.

[0069] The risk calculation module 202 is used to obtain the status information of other ships when the grid sensor senses other ships, the status information including speed and heading, and calculate the collision risk occurrence area based on the status information;

[0070] The avoidance module 203 is used to determine whether it is an avoidance ship or a holding ship when the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor. If it is a holding ship, the heading and speed are kept unchanged. If it is an avoidance ship, the status information of other ships is input into the anti-collision model to calculate the avoidance decision. After the decision layer executes the avoidance decision, the status information of itself and other ships is updated in the environmental information, and the execution is repeated until the distance between the ship itself and other ships is greater than a threshold or the distance between the ship itself and other ships is less than a threshold and there is no collision risk.

[0071] The advantages of the device are as follows:

[0072] Simplify input and calculation: Introduce virtual grid sensors as input. This method quantifies the dimension of the observed state, avoids the problem of too high a dimension of the state space, reduces computational complexity, and improves the execution efficiency of the model. The model has a simple structure and low input dimension, which enables it to have fast real-time execution capabilities in complex environments. In particular, by setting the risk collision area, the complexity of the state space is reduced and the convergence speed of the reinforcement learning algorithm is improved. By designing a simplified navigation situation judgment method, the model can distinguish between avoiding ships and keeping ships without adding additional model complexity, which improves the generalization ability and practicality of the model.

[0073] Based on the same idea, the embodiment of this specification also provides a ship automatic avoidance and anti-collision device, such as Figure 3 shown.

[0074] The ship automatic avoidance and collision prevention device may be the terminal device or server provided in the above embodiment.

[0075] The ship automatic avoidance and anti-collision device may have relatively large differences due to different configurations or performances, and may include one or more processors 301 and memory 302, and the memory 302 may store one or more storage applications or data. Among them, the memory 302 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit, and may further include a read-only storage unit. The application stored in the memory 302 may include one or more program modules (not shown in the figure), such program modules include but are not limited to: an operating system, one or more application programs, other program modules and program data, each of these examples or some combination may include the implementation of a network environment. Furthermore, the processor 301 can be configured to communicate with the memory 302 to execute a series of computer executable instructions in the memory 302 on the ship automatic avoidance and anti-collision device. The ship automatic avoidance and collision prevention device may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more I / O interfaces (input and output interfaces) 305, one or more external devices 306 (such as keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that enable a user to interact with the device, and / or communicate with any device that enables the device to communicate with one or more other computing devices (such as routers, modems, etc.). Such communication may be performed through the I / O interface 305. In addition, the device may also communicate with one or more networks (such as a local area network (LAN)) through the wired or wireless interface 304.

[0076] Specifically in this embodiment, the ship automatic avoidance and collision prevention device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the ship automatic avoidance and collision prevention device, and the one or more programs are configured to be executed by one or more processors, including the following computer executable instructions:

[0077] An anti-collision model is constructed based on deep reinforcement learning training to make avoidance decisions. The anti-collision model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, which is used to detect environmental information of the ship itself and its surroundings as input to the anti-collision model. The decision layer includes a Q-Learning algorithm, which is used to determine the optimal avoidance decision according to the input of the input layer. The action layer is used to adjust the speed and heading of the ship according to the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is a reward for the ship when it is sailing towards the target point, the heading deviation reward is a reward for the ship when its heading is sailing towards the target point, the collision avoidance reward is a reward for the ship when it avoids a collision, and the rule reward is a reward for the ship when it turns right to avoid a collision.

[0078] When the grid sensor senses other ships, status information of the other ships is obtained, the status information including speed and heading, and a collision risk occurrence area is calculated based on the status information;

[0079] When the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor, it is determined whether the ship itself is an avoiding ship or a holding ship. If it is a holding ship, the heading and speed are kept unchanged. If it is an avoiding ship, the status information of other ships is input into the anti-collision model to calculate the avoidance decision. After the decision layer executes the avoidance decision, the status information of itself and other ships is updated in the environmental information, and the execution is repeated until the distance between the ship itself and other ships is greater than a threshold or the distance between the ship itself and other ships is less than a threshold and there is no collision risk.

[0080] Based on the same idea, the exemplary embodiment of the present disclosure also provides a computer-readable storage medium on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above "Ship Automatic Avoidance and Anti-Collision Method" section of the present specification.

[0081] refer to Figure 4As shown, a program product 400 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, system or device.

[0082] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0083] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, system, or device.

[0084] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0085] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0086] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal system, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0087] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0088] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0089] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A method for automatic ship avoidance and collision prevention, characterized in that: The method comprises: An anti-collision model is constructed based on deep reinforcement learning training to make avoidance decisions. The anti-collision model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, which is used to detect environmental information of the ship itself and its surroundings as input to the anti-collision model. The decision layer includes a Q-Learning algorithm, which is used to determine the optimal avoidance decision according to the input of the input layer. The action layer is used to adjust the speed and heading of the ship according to the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is a reward for the ship when it is sailing towards the target point, the heading deviation reward is a reward for the ship when its heading is sailing towards the target point, the collision avoidance reward is a reward for the ship when it avoids a collision, and the rule reward is a reward for the ship when it turns right to avoid a collision. When the grid sensor senses other ships, status information of the other ships is obtained, the status information including speed and heading, and a collision risk occurrence area is calculated based on the status information; When the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor, it is determined whether the ship itself is an avoiding ship or a holding ship. If it is a holding ship, the heading and speed are kept unchanged. If it is an avoiding ship, the status information of other ships is input into the anti-collision model to calculate the avoidance decision. After the decision layer executes the avoidance decision, the status information of itself and other ships is updated in the environmental information, and the execution is repeated until the distance between the ship itself and other ships is greater than a threshold or the distance between the ship itself and other ships is less than a threshold and there is no collision risk.

2. A method for automatic ship avoidance and collision prevention according to claim 1, characterized in that: When calculating the collision risk occurrence area, the calculation is as follows: Where OS is the ship itself, TS is other ships, a is the safe passing distance, a = arcsin (r S / d), d is the distance between own ship and target ship, V O is the speed of own ship, V T is the speed of the target ship, A Z is the azimuth from own ship to target ship, C T is the heading of the target ship. When the ship is on the collision course, C O The relative motion is as follows: V R and C R The relative speed and course of the target ship relative to the own ship are calculated, and then the estimated approach time and distance between the two ships are calculated: DCPA=d|sin(C R -A Z +π)|; DCPA is the distance at the time of approach, and TCPA is the time of approach.

3. A method for automatic ship avoidance and collision prevention according to claim 1, characterized in that: When the collision risk area overlaps with the sensing area of ​​the grid sensor, the component of the state vector with the largest overlapping area is set to 1, and the setting without overlap is set to 0. If the sensing area of ​​the grid sensor overlaps with the collision risk area, the state vector closest to the ship itself is set to 1.

4. A method for automatic ship avoidance and collision prevention according to claim 1, characterized in that: The target driving reward and the heading deviation reward work in the target driving mode. When the collision risk occurrence area does not overlap with the sensing area of ​​the grid sensor, the ship itself maintains its heading towards the target point.

5. The method for automatic ship avoidance and collision prevention according to claim 1, characterized in that: The collision avoidance reward and rule reward work in the collision avoidance mode. When the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor, the Q value of the decision layer is updated based on the collision avoidance reward and rule reward.

6. A ship automatic avoidance and anti-collision device, characterized in that: The device comprises: A model generation module, used to build an anti-collision model based on deep reinforcement learning training to make avoidance decisions, the anti-collision model includes an input layer, a decision layer, an action layer, and a reward layer, the input layer includes a grid sensor, the grid sensor is used to detect the environmental information of the ship itself and the surroundings as the input of the anti-collision model, the decision layer includes a Q-Learning algorithm, the Q-Learning algorithm is used to determine the optimal avoidance decision according to the input of the input layer, the action layer is used to adjust the speed and heading of the ship according to the avoidance decision, and the reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward, the target-driven reward is a reward for the ship when it sails towards the target point, the heading deviation reward is a reward for the ship when its heading is sailing towards the target point, the collision avoidance reward is a reward for the ship when it avoids a collision, and the rule reward is a reward for the ship when it turns right to avoid a collision; A risk calculation module, used for obtaining status information of other ships when the grid sensor senses other ships, the status information including speed and heading, and calculating a collision risk occurrence area based on the status information; The avoidance module is used to determine whether the ship itself is an avoidance ship or a holding ship when the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor. If it is a holding ship, the heading and speed are kept unchanged. If it is an avoidance ship, the status information of other ships is input into the anti-collision model to calculate the avoidance decision. After the decision layer executes the avoidance decision, the status information of the ship itself and other ships is updated in the environmental information, and the execution is repeated until the distance between the ship itself and other ships is greater than a threshold or the distance between the ship itself and other ships is less than a threshold and there is no collision risk.

7. A ship automatic avoidance and anti-collision device, characterized in that: include: processor; and a memory arranged to store computer executable instructions which, when executed, cause the processor to: An anti-collision model is constructed based on deep reinforcement learning training to make avoidance decisions. The anti-collision model includes an input layer, a decision layer, an action layer, and a reward layer. The input layer includes a grid sensor, which is used to detect environmental information of the ship itself and its surroundings as input to the anti-collision model. The decision layer includes a Q-Learning algorithm, which is used to determine the optimal avoidance decision according to the input of the input layer. The action layer is used to adjust the speed and heading of the ship according to the avoidance decision. The reward layer is used to update the output of the decision layer, which includes a target-driven reward, a heading deviation reward, a collision avoidance reward, and a rule reward. The target-driven reward is a reward for the ship when it is sailing towards the target point, the heading deviation reward is a reward for the ship when its heading is sailing towards the target point, the collision avoidance reward is a reward for the ship when it avoids a collision, and the rule reward is a reward for the ship when it turns right to avoid a collision. When the grid sensor senses other ships, status information of the other ships is obtained, the status information including speed and heading, and a collision risk occurrence area is calculated based on the status information; When the collision risk occurrence area overlaps with the sensing area of ​​the grid sensor, it is determined whether the ship itself is an avoiding ship or a holding ship. If it is a holding ship, the heading and speed are kept unchanged. If it is an avoiding ship, the status information of other ships is input into the anti-collision model to calculate the avoidance decision. After the decision layer executes the avoidance decision, the status information of itself and other ships is updated in the environmental information, and the execution is repeated until the distance between the ship itself and other ships is greater than a threshold or the distance between the ship itself and other ships is less than a threshold and there is no collision risk.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for automatic ship avoidance and collision prevention as claimed in any one of claims 1 to 5 is implemented.

Images