A ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints

By identifying different encounter scenarios and designing corresponding decision networks and reward function combinations, and combining DQN algorithm to train ship collision avoidance decision networks, the problem that unified reward function in the existing technology is difficult to adapt to collision avoidance rules, and realizes that in different scenarios Effective collision avoidance and compliance with rules.

CN114895673BActive Publication Date: 2024-11-22WUHAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210446491.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-26
Publication Date
2024-11-22
Estimated Expiration
2042-04-26

AI Technical Summary

Technical Problem

The existing ship collision avoidance methods based on reinforcement learning lack a comprehensive explanation of collision avoidance rules and good ship art. The unified reward function is difficult to adapt to the differences in different encounter scenarios, resulting in the decision network being unable to fully meet the requirements of collision avoidance rules.

Method used

By identifying different encounter scenarios, designing corresponding decision networks and reward function combinations, building a hierarchical reward function system, and combining DQN algorithm to train decision networks, ensuring that ships abide by collision avoidance rules and optimize avoidance actions in different encounter scenarios.

Benefits of technology

It realizes a concession plan that is more in line with collision avoidance rules in different encounter scenarios, improves the practicality of the algorithm and navigation safety, and ensures that the ship can effectively avoid collisions and abides by sea collision avoidance rules.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114895673B_ABST
    Figure CN114895673B_ABST
Patent Text Reader

Abstract

The present invention discloses a ship collision avoidance decision method based on deep reinforcement learning under rule constraints. The method takes into account the ship's maneuvering characteristics, introduces an MMG model, and infers the real-time navigation information of the ship through the model. Then, a state space and action space definition method that considers different scenarios and driving experience is designed, and a collision avoidance decision network construction and training framework is proposed. In addition, in order to fully consider the rules, a set of hierarchical reward function combinations are designed and used for the training of the later decision network. Subsequently, by introducing the DQN algorithm, a collision avoidance decision network training model is constructed, and after a large amount of training, a collision avoidance decision network that can be applied to different scenarios is obtained. Finally, in order to verify the applicability of the proposed method, a variety of simulation test scenarios are designed for comprehensive performance evaluation. The method enables ships to avoid collisions safely in complex environments while ensuring that they comply with the requirements of rules and good seamanship.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of ship collision avoidance decision-making, and specifically to a ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints. Background Art

[0002] In recent years, due to the development of theory and technology, the research on unmanned ships has been greatly developed. At the same time, some unmanned ships have been used in many maritime tasks, including environmental monitoring, marine transportation, coastal surveys and remote sensing. However, with the increasing complexity of the application scenarios of unmanned ships, the risks faced by unmanned ships are also increasing, which increases the possibility of maritime accidents involving unmanned ships. Among all kinds of maritime accidents, ship collision is the most common type of accident, and collision accidents often cause serious casualties, huge property losses and environmental pollution. Therefore, intelligent collision avoidance technology has become a core technology in the research of unmanned ships.

[0003] At present, there have been many studies on intelligent collision avoidance algorithms for ships. For example, artificial potential field method, speed barrier algorithm, dynamic window method and heuristic algorithm have been widely used in previous studies. At the same time, the progress of artificial intelligence technology, especially reinforcement learning, has provided a new solution to the problem of ship collision avoidance due to its obvious advantages in decision-making problems. In addition, in the process of ship collision avoidance, in order to maintain coordination with other ship operations, all ships must comply with the International Regulations for Preventing Collisions at Sea (hereinafter referred to as "Collision Regulations") and the requirements of good seamanship. Therefore, in actual navigation conditions, collision avoidance rules and good seamanship are important factors to be considered in collision avoidance algorithms.

[0004] Since reinforcement learning methods have good self-learning ability, they are widely used to solve complex sequential optimization decision problems (Sutton and Barto, 2017). At the same time, reinforcement learning methods have a strong understanding of unknown environments and have great potential in solving highly complex ship collision avoidance problems. In addition, compared with existing collision avoidance methods, reinforcement learning methods have obvious advantages in anti-interference ability and decision-making timeliness.

[0005] However, a large number of existing ship collision avoidance studies based on reinforcement learning methods focus on model optimization, and the algorithms lack a comprehensive explanation of collision avoidance rules and good seamanship. More precisely, these studies use a unified reward function to evaluate the reward value of the avoidance action of the agent (ship) in different encounter scenarios, without fully considering that the collision avoidance requirements of the collision avoidance rules for different encounter scenarios are actually significantly different, and this difference is not well reflected in the reward function setting process. In addition, in the current research on ship collision avoidance methods based on reinforcement learning, researchers often set up a unified decision network and train the decision network using a set of reward functions. After the training is completed, the network is used as the ship avoidance decision network for all encounter scenarios. However, as mentioned above, due to the differences in the avoidance requirements of the collision avoidance rules for different encounter scenarios, this ship collision avoidance decision framework that uses a unified reward function to train a decision network cannot fully meet the ship avoidance requirements of the collision avoidance rules for different encounter scenarios. Summary of the invention

[0006] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a ship collision avoidance decision method based on deep reinforcement learning under rule constraints, so as to solve at least one of the above-mentioned technical problems.

[0007] The present invention is achieved through the following technical solutions:

[0008] A ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints, comprising:

[0009] Identify encounter scenarios;

[0010] For the identified encounter scenario, using a decision network corresponding to the encounter scenario to execute a ship collision avoidance decision, wherein the decision network corresponding to the encounter scenario is trained by a reward function corresponding to the encounter scenario, and the reward function is set in combination with a collision avoidance rule for the encounter scenario;

[0011] Output the collision avoidance action that should be performed.

[0012] The above technical solution provides a new decision-making framework by dividing different encounter types, provides a hierarchical reward function setting method, and defines corresponding reward function combinations for different encounter scenarios, fully considering ship navigation safety, collision avoidance rules and good seamanship, thus laying the foundation for the practical application of reinforcement learning methods in ship intelligent collision avoidance.

[0013] As a further technical solution, all encounter scenarios are divided into N types according to the action constraints of collision avoidance rules for different encounter scenarios, and a corresponding reward function is set for each encounter type to form N reward functions; wherein each reward function is used to train a decision network.

[0014] The above technical solution divides typical ship encounter scenarios, considers the corresponding requirements of collision avoidance rules for different encounter scenarios, sets reward functions and network structures in a targeted manner, and the reward function in each scenario is only used for training the decision network for that scenario. The decision network trained under this design mode will fully consider the specific requirements of collision avoidance rules in different encounter scenarios, so as to formulate avoidance plans that are more in line with collision avoidance rules according to different encounter scenarios, and the algorithm is more practical.

[0015] As a further technical solution, the method further includes: constructing a ship motion model for establishing a conversion relationship between the current state of the ship and the state at the next moment.

[0016] The above ship motion model describes the corresponding relationship between the change of the ship's rudder angle and the motion parameters. The rudder angle information of the ship's maneuvering is the input of the model, and the output information of the model is the real-time motion parameters of the ship (longitudinal speed v, lateral speed u, yaw angular velocity r).

[0017] As a further technical solution, three state spaces are defined according to different encounter scenarios, namely, the state spaces in multi-ship encounter, two-ship encounter and static obstacle avoidance scenarios; among them, the state space in the multi-ship encounter scenario consists of 4 parts and contains 22 elements, the state space in the two-ship encounter scenario consists of 3 parts and contains 8 elements, and the state space in the static obstacle avoidance scenario consists of 3 parts and contains 6 elements.

[0018] The above technical solution constructs a new state space by considering the characteristics of different encounter scenarios, and constructs and trains decision networks under different encounter scenarios on this basis.

[0019] As a further technical solution, considering the maneuvering habits of seafarers in actual collision avoidance, a discrete action space is constructed based on the rudder angle. The action space is a vector ranging from -35° to 35° and containing 11 elements, each element representing an alternative rudder angle of the ship in the process of avoiding obstacles.

[0020] The selection of these elements in the above technical solution fully considers the sailing experience, because the driver usually chooses to execute full left or right rudder only in very urgent situations, while under normal circumstances the driver generally tends to execute a moderate rudder angle.

[0021] By constructing the state space and action space, a mapping relationship between state and action is constructed, that is, when the ship observes a state from the environment, it can perform a collision avoidance action that maximizes future rewards.

[0022] As a further technical solution, after the ship recognizes the encounter scene from the environment, the spatial state observed by the ship is used as the input of the decision network corresponding to the encounter scene, and the future reward corresponding to each action in the action space is used as the output of the decision network; then the ship selects the action with the highest future reward to execute.

[0023] As a further technical solution, the reward function is arranged in layers, wherein the first layer is the base layer, and the reward function defined in the base layer takes into account the safety of the ship in the collision avoidance process, general ship handling experience and good seamanship, and all ship encounter scenarios need to execute the reward function in the base layer; the second layer is the rule layer, and the reward function defined in the rule layer is designed according to the avoidance requirements of the collision avoidance rules for different encounter scenarios. The reward function in the rule layer does not need to be executed by all ships, but only needs to select the corresponding reward function according to the actual encounter scenario for execution; the reward function value finally obtained by the ship is the sum of the reward functions of the base layer and the rule layer.

[0024] The reward is a quantitative evaluation of the actions performed by the ship. The ship will continuously optimize its action selection based on the reward feedback it receives from the environment, and can eventually complete the task as expected. The above technical solution enables the ship to avoid collisions with other ships by controlling its own rudder angle changes while meeting the collision avoidance rules, and deviate from the original course as little as possible.

[0025] The above technical solution evaluates and constrains the avoidance behavior of ships in different encounter states through a hierarchical reward function, thereby ensuring that ships can strictly abide by the corresponding restrictions of collision avoidance rules in different encounter scenarios on the basis of ensuring navigation safety.

[0026] As a further technical solution, the reward functions defined in the base layer include a target reward function, a forward reward function, a collision reward function, a rudder angle reward function and a yaw reward function.

[0027] As a further technical solution, the reward functions defined in the rule layer include a straight-line ship reward function, a give-way ship reward function, an encounter scenario reward function, a cross-encounter scenario reward function and an overtaking scenario reward function.

[0028] As a further technical solution, based on the combination of hierarchical reward functions, the DQN algorithm is introduced to train the decision network.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The present invention aims to provide an intelligent collision avoidance method that complies with the ship collision avoidance rules to ensure the navigation safety of unmanned ships, and accordingly proposes a collision avoidance decision framework based on deep reinforcement learning. First, a state space and collision avoidance decision network structure design method considering the characteristics of different encounter scenarios is proposed, and it is used as the basis of the collision avoidance framework. Then, a set of hierarchical reward functions is established to evaluate and constrain the avoidance behavior of ships under different encounter states, thereby ensuring that ships can strictly abide by the corresponding restrictions of collision avoidance rules in different encounter scenarios on the basis of ensuring navigation safety. Driven by the combination of hierarchical reward functions, the DQN algorithm is further introduced to train the proposed collision avoidance decision framework. After the training process is completed, a number of simulation experiments are carried out to evaluate the practicality of the decision framework. The results show that the framework can effectively complete the collision avoidance task in different ship encounter scenarios.

[0031] (2) The present invention fully considers the requirements of the rules for ship avoidance actions in different encounter scenarios and the maneuverability characteristics of the ship, and designs corresponding reward function combinations for different encounter scenarios. This design method will take into account the special requirements of the collision avoidance rules for avoidance actions in different encounter scenarios in a more targeted manner. At the same time, by improving the construction, training and decision-making methods of the collision avoidance decision network, the decision network can provide avoidance action plans that are more in line with the collision avoidance rules according to different encounter scenarios, thereby solving the problem in current related research that a single reward function combination is difficult to adapt to the differences in the avoidance action requirements of the collision avoidance rules for different encounter scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the comparison before and after the improvement of the collision avoidance algorithm training framework according to an embodiment of the present invention.

[0033] Figure 2 Schematic diagram showing the comparison before and after the improvement of the collision avoidance algorithm decision framework according to an embodiment of the present invention.

[0034] Figure 3 Schematic diagram of a geodetic coordinate system and a ship-borne coordinate system according to an embodiment of the present invention.

[0035] Figure 4 This is a schematic diagram of the application principle of the deep reinforcement learning algorithm according to an embodiment of the present invention in the ship collision avoidance problem.

[0036] Figure 5 It is a schematic diagram of target ship division and state space determination based on collision avoidance rules according to an embodiment of the present invention.

[0037] Figure 6 Schematic diagram of a state space positioning method in different encounter scenarios according to an embodiment of the present invention.

[0038] Figure 7The following is an action selection process of a ship (agent) according to an embodiment of the present invention.

[0039] Figure 8 Schematic diagram of reward function composition and reward function combination used in different encounter scenarios according to an embodiment of the present invention.

[0040] Fig. 9 4 is a framework diagram of a collision avoidance algorithm based on a DQN algorithm according to an embodiment of the present invention.

[0041] Fig.10 Schematic diagram of comparison before and after improvement of a collision avoidance training framework with seven encounter scenarios according to an embodiment of the present invention.

[0042] Fig.11 It is a schematic diagram of comparison before and after improvement of a collision avoidance decision framework with seven encounter scenarios according to an embodiment of the present invention.

[0043] Fig.12 Schematic diagram of the change of each fragment reward function value with the number of training times according to an embodiment of the present invention.

[0044] Fig.13 It is a schematic diagram of simulation results of avoiding static obstacles according to an embodiment of the present invention.

[0045] Fig.14 Schematic diagram of two-ship collision avoidance simulation results according to an embodiment of the present invention.

[0046] Fig.15 Schematic diagram of multi-ship collision avoidance simulation results according to an embodiment of the present invention.

[0047] Fig.16 This is a schematic diagram of a simulation result of collision avoidance between two ships in a scenario where the straight-line ship needs to take action according to an embodiment of the present invention.

[0048] Fig.17 It is a schematic diagram of projection of the ship speed in the forward direction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] The present invention constructs a ship collision avoidance decision framework based on deep reinforcement learning, taking into full consideration the constraints of collision avoidance rules and the dynamic characteristics of ships. At the same time, the deep DQN algorithm is used to train the neural network. In the proposed collision avoidance algorithm framework, according to the different requirements of the collision avoidance rules for ship collision avoidance actions, the ship encounter scenarios are divided into different types, and each encounter type defines a corresponding decision network structure and an exclusive reward function combination. In addition, the present invention also verifies the proposed collision avoidance framework using data from a real ship.

[0051] In general, the present invention proposes a ship collision avoidance decision-making framework based on deep reinforcement learning methods. This framework carefully considers the requirements of the collision avoidance rules for ship avoidance actions in different encounter scenarios, and also considers the maneuverability characteristics of the ship. Secondly, in terms of algorithm details, a set of layered reward functions is designed. Compared with previous related studies, the present invention does not use a unified reward function combination, but designs corresponding reward function combinations for different encounter scenarios. This design method will be more targeted and take into account the special requirements of the collision avoidance rules for avoidance actions in different encounter scenarios. Finally, the present invention also proposes a set of improved collision avoidance decision network construction, training and decision-making methods, so that the decision network can provide avoidance action plans that are more in line with the collision avoidance rules according to different encounter scenarios, such as Figure 1 and Figure 2 As shown, n is a finite constant. The collision avoidance decision framework proposed in the present invention will solve the problem that a single reward function combination is difficult to adapt to the differences in the avoidance action requirements of collision avoidance rules for different encounter scenarios in current related research.

[0052] In order to describe the motion of the ship, a ship motion coordinate system is established, such as Figure 3 As shown in the figure, the XOY coordinate system is the geodetic coordinate system, and the xoy coordinate system is the ship-borne coordinate system. The origin of the xoy coordinate system is "o", which is the center of gravity of the ship; "O" is the origin of the XOY coordinate system; X 0 , Y 0 are the projections of the ship’s center of gravity on the X-axis and Y-axis respectively; ψ represents the ship’s heading, and δ represents the ship’s rudder angle.

[0053] The position and speed of the ship in the XOY and xoy coordinate systems can be converted to each other using formula (1) and formula (2):

[0054]

[0055] Where A is the transformation matrix, [X,Y] and [x,y] are the coordinates of the ship in the coordinate systems XOY and xoy respectively, [X 0 ,Y 0 ] is the coordinate value of the origin "o" of the coordinate system xoy in the XOY coordinate system.

[0056] Ships will be greatly affected by hydrodynamic forces during navigation. In order to accurately describe the motion trajectory of the ship, the hydrodynamic characteristics of the ship need to be considered when calculating and updating the ship's position information. Since the vertical motion of the ship has little effect on the navigation trajectory of the ship, the present invention mainly studies the motion of the ship in the horizontal dimension. Let the longitudinal velocity in the xoy coordinate system be v, the lateral velocity be u, and the yaw angular velocity be r. The motion model can be expressed as formula (4) using the MMG model.

[0057]

[0058] m is the mass of the hull, m x and m y is the additional mass of the hull along the x-axis and the additional mass along the y-axis. H , Y H , X P , Y P , X R , Y R are the external forces on the hull, propeller and rudder on the x-axis and y-axis respectively. ZZ and J ZZ is the moment of inertia about the z-axis. N H and N R is the yaw moment about the hull and rudder in the z-axis.

[0059] This model describes the corresponding relationship between the change of the ship's rudder angle and the motion parameters. The rudder angle information of the ship's maneuvering is the input of the model, and the output information of the model is the real-time motion parameters of the ship (longitudinal speed v, lateral speed u, yaw angular velocity r). Therefore, the position and heading information of the ship in the xoy coordinate system can be calculated in real time by formula (5). Among them, x(0), y(0) and ψ(0) are the initial position and heading of the ship in the xoy coordinate system. Combining formula (2), formula (4) and formula (5), the position, heading and speed of the ship in the XOY coordinate system can be calculated in real time.

[0060]

[0061] The ship motion model establishes the conversion relationship between the current state of the ship and the state at the next moment. Therefore, the ship collision avoidance problem can be defined as a sequential decision problem, which can be modeled using a Markov decision process (MDP).

[0062] like Figure 4 As shown, the ship (agent) starts from the starting state s 0 Start, then in the strategy π θ (a|s) chooses a method that maximizes future returns G = Σ∞k = 0γk r k+1 Action 0 ∈A(s 0 ). A(s 0 ) is the agent in state s 0 The set of actions that can be selected when π θ (a|s) represents when s t =s when a t = the probability of a, that is, when the agent is in state s, the probability of choosing to perform action a, γ is the loss coefficient, γ∈[0,1]. It should be noted that the strategy π used here θ (a|s) is an ε-greedy strategy that balances the relationship between development and exploration. "Development" means that the agent chooses to perform an action that can maximize the reward value, while "exploration" means that the agent randomly performs an action that may be selected. The addition of this "exploration" strategy can prevent the algorithm from falling into a local optimum. When the agent completes the action selection, the agent will execute this action and enter a new state s 1 , and obtain a reward feedback value r from the environment 1 Then the strategy π θ The parameters θ in (a|s) will be updated according to the environment reward value. Then, the agent will continue to execute the above process until it reaches the terminal state s n Through continuous interaction with the environment, the agent will converge to a target strategy. After converging to this target strategy, the agent will no longer use the ε-greedy strategy when executing actions, but will directly choose actions that maximize the value of the reward function, that is, only use the "exploitation" strategy.

[0063] The state space is the result of the agent's observation of the environment and is also the basis for the agent to choose to perform actions. According to different encounter scenarios, the present invention proposes three methods for defining the state space.

[0064] (1) Multiple ships will encounter

[0065] According to the collision avoidance rules, there are three types of encounter situations between the two ships in collision avoidance, depending on the direction of the approaching ship: overtaking, meeting and crossing. For approaching ships from different directions, the avoidance responsibility of the own ship (the intelligent agent in this invention) can be used Figure 5It should be noted that the coordinate system used here is the xoy coordinate system (ship coordinate system), that is, these directions are relative to the agent itself, the origin of the coordinate system is the center of gravity of the agent, and the detection distance of the agent is selected as 6 nautical miles based on general navigation experience. According to the collision avoidance rules, the agent has different avoidance responsibilities and action requirements for ships in different areas. When there are multiple ships near the agent, this invention will use the method in the paper (Zhao and Roh, 2019) to select the information of the closest ship to the ship in each area to form the input state S TM . S TM The definition is shown in formula (6):

[0066] S TM =[d Ti , β Ti , ψ Ti , v Ti ] (6)

[0067] Among them, d Ti is the distance between the agent and the nearest ship in each area, β Ti is the relative position of the agent and the nearest ship in each area, ψ Ti is the heading of the nearest ship in each area, v Ti is the speed of the nearest ship in each area, and i represents the index number of each area. It should be noted that d Ti and β Ti Two sets of parameters are relative to the ship coordinate system, while the other two sets of parameters are relative to the earth coordinate system. In addition, when there is no incoming ship in one of the four areas divided above, the four elements of the target ship parameters in this area will be assigned 0, so that this area will not have any impact on the execution of future actions.

[0068] However, in addition to the dynamic target ship, the intelligent agent may also encounter static obstacles during navigation. Since the collision avoidance rules do not restrict the actions of the ship when avoiding static obstacles, that is, for static obstacles in any area that pose a collision hazard to the ship, the ship can adopt any action that is most conducive to avoiding collision, and the choice of ship avoidance action will not be affected by the different areas where the static obstacles are located. Therefore, the present invention selects the state S of the obstacle closest to the intelligent agent among all static obstacles that pose a collision hazard to the intelligent agent. OM As one of the input information of the algorithm. OM The definition of is shown in formula (7).

[0069] S OM =[d O , β O ] (7)

[0070] Among them, d O is the distance between the obstacle and the agent, β O is the position of the obstacle relative to the agent.

[0071] Excluding the above defined ship arrival status S TM and the static obstacle state S OM , the state S of the agent itself AM and the terminal state S DM will also affect the execution of the final agent action. AM and S DM They are defined as formula (8) and formula (9) respectively.

[0072] S AM =[ψ A ,v A ] (8)

[0073] S DM =[d D ,β D ] (9)

[0074] Among them, ψ A is the heading of the agent, v A is the speed of the agent, d D is the distance between the agent and the endpoint, β D is the position of the endpoint relative to the agent.

[0075] Therefore, the state space S in the multi-ship encounter scenario is M S TM , S OM , S AM and S DM It consists of four parts and contains a total of 22 elements.

[0076] (2) The two ships will meet

[0077] The state space definition method in the two-ship encounter scenario is similar to the definition method in the multi-ship encounter scenario. However, because the presence of static obstacles may affect the normal execution of the collision avoidance rules by ships, and the specific provisions in the collision avoidance rules are mainly formulated for the two-ship scenario, in order for the intelligent agent to fully execute the collision avoidance requirements of the collision avoidance rules, the present invention will not consider the presence of static obstacles in the state space definition process in the two-ship encounter scenario. When there is a static obstacle effect, the definition method of its state space can refer to the multi-ship encounter scenario. Therefore, the state space S in the two-ship encounter scenario T It consists of three parts: target ship state S TT , agent state S AT , and the terminal state S DT . S TTThe definition is as shown in formula (10), S AT and S DT The definition of is the same as that in the multi-ship encounter scenario. Therefore, the state space in the two-ship encounter scenario contains a total of 8 elements.

[0078] S TT =[d T ,β T ,ψ T ,v T ] (10)

[0079] Among them, d T is the distance between the agent and the target ship, β T is the relative position of the incoming ship to the agent, ψ T is the heading of the incoming ship, v T It is the speed of the approaching ship.

[0080] (3) Static obstacle avoidance

[0081] In this scenario, the collision avoidance rules will not impose any restrictions on the collision avoidance actions of the ship. The ship only needs to take effective avoidance actions based on the information of obstacles and the destination to be reached. Therefore, the state space S in this scenario S It consists of three parts: static obstacle state S OS , agent state S AS and the terminal state S DS The definition of these states is the same as that in the multi-ship encounter scenario, so the state space corresponding to this scenario includes a total of 6 elements.

[0082] In general, the state space in the multi-ship encounter scenario consists of four parts, containing a total of 22 elements. The two-ship encounter scenario and the static obstacle avoidance scenario each consist of three parts, containing 8 and 6 elements respectively. The definition method of the state space can be summarized as follows: Figure 6 As shown. It should be noted that the heading and speed information contained in the state space are values ​​relative to the geodetic coordinate system, while the distance and azimuth information are information relative to the agent, that is, values ​​in the ship coordinate system. These defined state spaces will be input into the collision avoidance decision network as input information.

[0083] During the collision avoidance process, the driver can avoid obstacles (including dynamic ships and static obstacles) by changing the course or speed. However, due to the huge inertia of the ship, the method of changing the speed often cannot immediately achieve obvious results. Therefore, the driver generally chooses to keep the speed unchanged and only change the course to avoid collisions between ships. Because the driver can adjust the course of the ship by controlling the rudder angle, the present invention uses the rudder angle as the action space, and by considering the driver's operating habits, a discrete action space A is constructed. The specific definition method is shown in formula (11).

[0084] A=[-35°,-20°,-15°,-10°,-5°,0°,5°,10°,15°,20°,35°] (11)

[0085] Among them, A is a vector with 11 elements ranging from -35° to 35°, each element represents the rudder angle that the agent can choose when avoiding obstacles. The selection of these elements fully considers sailing experience, because drivers usually only choose to perform left or right full rudder in very urgent situations, and usually drivers tend to perform a moderate rudder angle.

[0086] The purpose of this study is to construct a mapping relationship between states and actions, that is, when the agent observes a state from the environment, the agent can perform a collision avoidance action that maximizes future rewards. Therefore, the input of the decision network is the state S observed by the agent. However, it should be noted that the output of the decision network is not the action that the agent will perform, but the future reward Q corresponding to each action in the action space. The agent will choose the action with the highest future reward to perform. The process from observation state input to action output is as follows: Figure 7 Therefore, the number of output neurons of the decision network is the same as the number of elements contained in the action space.

[0087] Reward is a quantitative evaluation of the actions performed by the agent. The agent will continuously optimize its action selection based on the reward feedback it receives from the environment, and can eventually complete the task as expected. This study aims to enable the agent to avoid collisions with other ships by controlling its own rudder angle changes while meeting the collision avoidance rules, and deviate from the original course as little as possible.

[0088] According to the rules for avoiding collisions at sea, the agent has different collision avoidance responsibilities for ships coming from different directions, and the collision avoidance actions that should be performed are also different. Therefore, the present invention constructs a set of layered reward functions. Among them, the first layer is the basic layer. The reward function defined in this layer will mainly consider the safety of the agent in the collision avoidance process, general ship handling experience and good seamanship. All ship encounter scenarios need to execute the reward function in this layer. The second layer of reward functions is defined as the rule layer. The reward function defined in this layer will mainly be designed for the avoidance requirements of the collision avoidance rules for different encounter scenarios. The agent does not need to execute all the reward functions in this layer, but only needs to select the corresponding reward function according to the actual encounter scenario. The reward function value finally obtained by the agent is the sum of the reward functions of the basic layer and the rule layer.

[0089] (1) Base layer

[0090] The main purpose of the reward function defined in the base layer is to drive the agent to find a safe and economical collision avoidance path in different encounter scenarios. That is, it only considers the requirements of collision avoidance safety and economy. The reward function layer includes 5 reward functions, namely, the target reward function R goal , forward reward function R advance , collision reward function R collision , rudder angle reward function R rudder and the yaw reward function R yaw The meaning and representation of each reward function are as follows.

[0091] Target reward function R goal The definition of is to guide the agent to continuously approach the end point. The reward function can be defined as formula (12):

[0092]

[0093] Among them, distance goal_t is the distance from the agent to the end point at time t, γ 0 , r goal and λ goal are all constants. When the agent approaches the end point, the reward function value R goal is a positive value. If the agent is far away from the end point, the reward function value R goal is a negative value. goal_t Less than γ 0 , it is considered that the agent has reached the end point and obtains a large reward value r goal .

[0094] In addition, in order to drive the agent forward, the projection of the agent's velocity in the forward direction must be positive. Therefore, the forward reward function Radvance It can be defined as the form of formula (13):

[0095]

[0096] Among them, R advance represents the forward reward function value, r advance is a constant, v advance_t is the velocity projection of the agent in the forward direction at time t. Velocity projection v advance_t The calculation method of Fig.17 and Table 3. When v advance_t When v is positive, the agent will receive a positive reward function. advance_t When it is negative, the agent will receive a small negative reward.

[0097] Collision reward function R collision It plays a very important role in motivating the intelligent agent to avoid obstacles smoothly. Its definition can be expressed as formula (14):

[0098]

[0099] Among them, distance obstacle_t is the minimum distance between the agent and surrounding obstacles, γ 1 and r collision is a constant. obstacle_t Less than γ 1 When the distance obstacle_t Not less than γ 1 , the agent will not receive any reward.

[0100] In addition, in order to avoid excessive yaw while maintaining a certain steering effect, the driver generally chooses a moderate steering angle to avoid obstacles. Therefore, the present invention also designs a steering angle reward function and a yaw reward function, and the specific definition forms are shown in formula (15) and formula (16) respectively.

[0101]

[0102] Among them, rudder angle_t is the action performed by the agent at time t, r rudder , μ 1 and μ 2 is a constant, S yaw_t is the yaw distance of the agent at time t, which can be calculated using formula (17), S yaw_max is the maximum tolerable yaw distance.

[0103] S yaw=|∫v×sin(θ 0 -θ 1 )dt| (17)

[0104] Where v is the real-time speed of the agent, θ 0 and θ 1 The calculation method of Fig.17 and Table 3.

[0105] Table 3v advance Calculation

[0106]

[0107] θ 0 is the ship's heading, which can be obtained from the ship's real-time information. 2 It is the angle between the ship's speed direction and the forward direction, and the value is a positive value.

[0108] (2) Rule layer

[0109] The collision avoidance rules explain the responsibilities of ships in different encounter scenarios and make detailed provisions for the collision avoidance actions that ships should take in different encounter scenarios. According to these requirements, when the encounter scenarios in which ships are located are different, the avoidance plans that ships should implement are also very different. However, in previous studies, when designing reward functions related to collision avoidance rules, the differences in the evaluation of intelligent agent actions in different avoidance responsibilities and encounter scenarios are often not fully considered. In addition, the biggest difference between ship collision avoidance and collision avoidance research in other fields (robot collision avoidance, vehicle collision avoidance) is that ship collision avoidance needs to perform collision avoidance actions under the constraints of collision avoidance rules, so the implementation of collision avoidance rules will directly determine the practicality of the collision avoidance algorithm.

[0110] The present invention designs a set of differentiated reward functions for the maritime collision avoidance rules in the rule layer. In the design process of the reward function, the constraints of the rules on the collision avoidance actions of ships in different avoidance responsibilities and different encounter scenarios are considered at the same time. These reward functions can be generally divided into five different dimensions: a straight-line ship reward function, a give-way ship reward function, an encounter scenario reward function, a cross encounter scenario reward function, and an overtaking scenario reward function. It should be noted that the intelligent agent does not need to execute all the reward functions in the rule layer, but will execute the corresponding reward function according to its own avoidance responsibilities and the encounter scenario in which it is located. The five-dimensional reward functions in the rule layer are defined as follows:

[0111] (1) Direct Ship

[0112] By analyzing the description of the collision avoidance responsibility of straight-navigating ships in Article 17 of the Collision Avoidance Regulations, it can be found that the collision avoidance responsibility of straight-navigating ships will change with the development of the encounter situation. In the early stage of the formation of collision risk, straight-navigating ships do not need to take action. If the give-way ship does not take timely measures, the straight-navigating ship should take effective avoidance measures. Since the collision avoidance rules do not clearly define the actions that straight-navigating ships should perform during the collision avoidance stage, the present invention uses the reward function in the base layer to evaluate the actions of straight-navigating ships, and mainly considers the safety of collision avoidance and ship driving experience. However, since the rules clearly stipulate that in an intersection encounter situation, straight-navigating ships should avoid taking left-turn collision avoidance actions against ships on their left. Therefore, the present invention only formulates a reward function R for this situation in the design of the straight-navigating ship reward function. stand-on-crossing To constrain the behavior of straight-line ships, its definition is shown in formula (18).

[0113]

[0114] In an encounter scenario where the straight-line ship does not need to take action, the straight-line ship only needs to maintain its speed and heading, and does not need to determine the rudder angle to be executed, so the present invention does not define a separate reward function for the scenario where the straight-line ship does not need to perform an action.

[0115] (2) Give Way Vessel

[0116] According to Article 16 of the Collision Avoidance Rules, the collision avoidance rules mainly require the maneuvering of the give-way ship in three aspects: "early", "large" and "wide". Among them, "early" means that the give-way ship should take action as soon as possible; "large" means that the action taken by the give-way ship must be large and easy to be clearly identified by the approaching ship; "wide" means that the give-way ship must maintain a sufficient distance from the target ship during the collision avoidance process. According to the above requirements, the present invention has formulated three corresponding reward functions when designing the reward function for the give-way ship: R give-way1 , R give-way2 and R give-way3 .

[0117] R give-way1 =-r early ·Δt(19)

[0118]

[0119] Among them, r early , r substantial and r clear are three reward factors, Δt is the time interval between the agent detecting the collision risk and taking avoidance action; is the agent's heading change, is the smallest change in course that can be detected by the crew. is the threshold of acceptable course change, and the specific parameter value can be obtained by consulting experienced captains; D CPA is the shortest distance between the agent and the target ship, D col is the minimum safe distance that should be maintained between the agent and the target ship, D pre is the warning distance between the agent and the target ship, D pre Greater than D col , both of which are acquired based on sailing experience.

[0120] (3) Encounter

[0121] According to Article 14 of the Collision Avoidance Regulations, in the case of a head-on collision, the collision avoidance regulations require that the ship should perform two avoidance actions: "turn right" and "pass the other party's port side". Therefore, the present invention has formulated two corresponding reward functions when designing the encounter reward function: the right turn reward function R head-on1 and port through the reward function R head-on2 .

[0122]

[0123] Among them, r starboard and r pass is a constant, β CPA It is the position of the target ship relative to the agent when the two ships reach the closest encounter point.

[0124] (4) Cross-encounter

[0125] According to Article 15 of the Collision Avoidance Regulations, in the cross-over scenario, the collision avoidance regulations require that the ship should perform avoidance actions mainly to avoid crossing the bow of the other ship. Based on this limitation, the present invention mainly formulates the reward function R for avoiding crossing the bow of the oncoming ship when designing the cross-over reward function. crossing .

[0126]

[0127] Among them, r crossing is a constant, α CPA It is the position of the target ship relative to the agent when the two ships reach the closest encounter point.

[0128] (5) Overtaking

[0129] Article 13 of the Collision Avoidance Rules stipulates the avoidance responsibility of the overtaking ship, but does not clearly limit the avoidance actions that the overtaking ship should perform. Therefore, in the overtaking scenario, the present invention does not define a specific reward function, but directly uses the reward function in the base layer to constrain the actions of the agent, that is, it only ensures the safety and economy of the avoidance process, without explicitly limiting the avoidance actions performed by the agent.

[0130] As mentioned above, when training the decision network, the agent does not need to execute all the reward functions formulated above, but will execute the corresponding reward function according to its own avoidance responsibility and the encounter scenario.

[0131] According to its own avoidance responsibility and encounter situation, the present invention divides the scenes where the agent needs to perform avoidance actions into seven categories: there are five scenes where the agent is responsible for avoidance, namely, avoiding static obstacles, avoiding multiple ships, encountering, overtaking and crossing encounter; there are two scenes where the agent is responsible for direct navigation, namely, the agent is a direct navigation ship but needs to take action, and the agent is a direct navigation ship but needs to take action and is in a crossing encounter situation with the oncoming ship. The reward function combinations that should be executed in each of the above seven scenes are as follows: Figure 8 As shown. It should be noted that the present invention only classifies the encounter scenarios in which the agent needs to take evasive action, and does not consider the scenario in which the agent is a straight-line ship and does not need to perform evasive action. Because according to the collision avoidance rules, in this encounter scenario, the agent only needs to maintain its own speed and heading, and does not need to make special decisions on the actions it needs to perform. In addition, the quantitative judgment method of different encounter scenarios and the calculation method of the time point when the straight-line ship should start to perform evasive action can refer to the results of (He et al., 2017) and others.

[0132] The present invention introduces the deep Q network (DQN) algorithm to train the designed collision avoidance algorithm. The DQN algorithm was proposed in 2013. Based on its initial algorithm, scholars introduced the concept of target network in later research to improve the stability of training. Fig. 9 As shown, the DQN algorithm includes 5 parts: environment, loss function, evaluation network, target network and experience replay. It should be noted that the evaluation network and the target network are two neural networks with exactly the same structure, but the parameters in the evaluation network will be updated once the agent completes an action, that is, from the current state to the next state, while the parameters in the target network will only be updated once the agent completes a completed segment, that is, when the agent reaches the terminal state. Compared with traditional reinforcement learning methods, the DQN algorithm can be applied to continuous state space problems. In addition, the introduction of experience replay improves the data utilization efficiency and the learning speed of the algorithm. In order to consider the hydrodynamic characteristics of the ship, the present invention adds an MMG model to the algorithm framework, which can help the agent accurately calculate its state at each moment.

[0133] In each training step, the evaluation network will output an approximate action value vector Q(S, a; θ) based on the current state S of the agent, and then select an action a for the agent to perform in the current state by utilizing the ε-greedy strategy shown in formula (25).

[0134]

[0135] Then, the agent will perform action a and use the MMG model to calculate the state S' at the next moment, and obtain the reward value r of the environment feedback. Subsequently, the experience bar (S, a, r, S') will be saved in the experience playback module, and the agent will randomly sample an experience set (S containing ξ experiences) from the experience playback module. i ,a i ,r i ,S′ i ), the experience set will be used as a data set to evaluate the network parameter update. Then, the target network outputs an action value vector based on the sampled experience set, which will be used as the actual action value. The calculation process is shown in formula (26).

[0136] U i =r i +γmax a Q(S i ',a;θ')(26)

[0137] In order to update the network parameters, the mean square error is used to calculate the error between the actual action value and the approximate action value, and it is used as the loss function, as shown in formula (27).

[0138]

[0139] By using the stochastic gradient descent strategy of formula (28), the value of the loss function can be continuously optimized until the function converges to a stable value.

[0140]

[0141] After the above processes, the parameters of the evaluation network will be updated, and the state of the agent will transition from the current state S to the state S′ at the next moment.

[0142] When the agent completes a complete trajectory segment, that is, the agent reaches the terminal state, the parameters of the target network will be overwritten by the parameters of the evaluation network, so that the parameters of the target network are also updated, as shown in formula (29).

[0143] θ'=θ(29)

[0144] Algorithm: Collision avoidance algorithm based on DQN

[0145]

[0146] One advantage of the improved framework is that the characteristics of different encounter scenarios are fully considered in the definition of the state space. A scenario-based state space definition method is proposed. This method improves the applicability of the algorithm and also lays the foundation for building and training different network structures for different encounter scenarios.

[0147] Another advantage of the improved framework is that the algorithm divides all encounter scenarios into 7 types according to the action constraints of the collision avoidance rules for different encounter scenarios, and constructs a reward function combination for each encounter type to train the decision network. It should be noted that in the invention, 7 decision networks are constructed for the 7 different encounter types, and each decision network is trained using the corresponding reward function combination. This design fully considers the collision avoidance requirements of the collision avoidance rules for different encounter scenarios, solves the problem of insufficient consideration of the collision avoidance rules caused by the use of a set of reward functions to train decision networks and a single decision network to execute collision avoidance decisions, and ensures the compliance of collision avoidance actions with rules. See the block diagram for a comparison between the improved framework and the traditional reinforcement learning-based ship collision avoidance framework Fig.10 .

[0148] Once the training process is completed, the collision avoidance decision framework will obtain 7 collision avoidance decision networks suitable for different encounter types. When avoiding collisions, the agent will first determine the type of encounter it is in, and then select the corresponding neural network to make decisions. The decision network corresponding to each encounter type in the improved framework is trained using the reward function corresponding to the encounter type. Compared with using a single decision network to make decisions for all encounter scenarios, the avoidance plan formulated by the improved framework will more comprehensively consider the avoidance constraints of the collision avoidance rules for different encounter situations. The collision avoidance decision process of the agent is as follows: Fig.11 shown.

[0149] Example

[0150] The purpose of this embodiment is to train an intelligent agent so that it can perform safe and rule-compliant collision avoidance actions in different encounter scenarios. The deep neural network used in the embodiments of the present invention is constructed and trained using Tensorflowv2.2.0. These networks are all fully connected networks with two hidden layers, and the number of input neurons and output neurons of the network is equal to the number of elements contained in the state space and action space, respectively. The optimizer used in network training is the Adam optimizer, and the activation function used is the Relu activation function. The settings of other hyperparameters of the algorithm are shown in Table 1.

[0151] Table 1 Algorithm hyperparameter settings

[0152]

[0153] During the training process, the agent will calculate its real-time state information through the MMG model, and obtain corresponding reward feedback from the environment according to its state transformation. The MMG model parameters used in the experiment of the present invention are taken from the data of a real ship, and some of its main parameters are shown in Table 2. It should be noted that in order to reduce the amount of calculation, the state of the agent is updated every 20 seconds during the simulation experiment.

[0154] Table 2 Main ship parameters

[0155]

[0156] In order to effectively train the neural networks in the collision avoidance framework, each decision network was trained extensively in its corresponding encounter scenario. Fig.12 The results of network training in a complex multi-ship encounter scenario are described. Fig.12 In the figure, the curve depicts the change of the reward function value of each segment. The horizontal axis in the figure is the number of training times, and the vertical axis is the average reward function value with a dynamic window of 50. The network training process lasted for 5000 times. It can be seen from the figure that with the increase of the number of iterations, the average reward function also increases, and finally converges to a stable value, that is, the strategy π also converges to the final strategy.

[0157] In order to verify the effectiveness of the proposed collision avoidance framework, the present invention uses PyCharm to build various complex simulation cases. These include four scenarios: avoiding static obstacles, two-ship encounter, multiple-ship encounter, and a scenario where a straight-line ship needs to take action. It should be noted that the determination of avoidance responsibilities and the division of encounter scenarios in the simulation experiment are all based on the perspective of the agent. In addition, in order to increase the computing speed, the initial position and end point of the agent are pre-set within an acceptable range.

[0158] (1) Static obstacle avoidance

[0159] In this scenario, an isolated static obstacle is set in the center of the test area. The size of the test area is set to 6×6nmile, the midpoint of the lower boundary of the test area is the starting position of the agent, and the midpoint of the upper boundary of the test area is the end point of the agent's navigation task. When the agent reaches the fan-shaped area with a radius of 1000m around the end point, it is considered that the agent has completed the task. The initial heading of the agent is set to 0°, that is, if the agent wants to reach the end point, it must pass through the area where the obstacle is located. The minimum safe distance that needs to be maintained between the agent and the obstacle is set to 1000m.

[0160] like Fig.13In the ship trajectory shown, the agent finds a trajectory to the target area while safely avoiding obstacles, where 1 represents the rudder angle, 2 represents the heading, and 3 represents the distance. This trajectory is relatively smooth, without large changes in heading. In the simulation results, the agent performed an obvious avoidance operation at the starting position, so that its own heading no longer intersects with the obstacle. After sailing on the new heading for a period of time, in order to control the deviation distance, the agent executed a left rudder angle and gradually returned to its initial heading. As the agent continued to approach the target area, the agent executed another left turn and finally entered the target area. From Fig.13 From the avoidance information on the right, it can be found that the distance between the agent and the end point basically shows a linear decreasing trend with the increase of time. The distance between the agent and the obstacle experienced a process of first decreasing and then increasing. During the entire navigation process, the minimum distance between the agent and the obstacle was 2025m, which occurred at 44×20s. According to the analysis, this distance is slightly larger than the minimum safety distance threshold set previously, which increases the yaw cost of the agent to a certain extent. However, the analysis believes that this is because if the agent collides with the obstacle, the penalty value obtained by the agent will be much greater than the penalty value caused by yaw. Therefore, in order to ensure navigation safety, the agent tends to maintain a relatively large distance when bypassing obstacles. This phenomenon also exists in the simulation experiments of subsequent scenes.

[0161] (2) The encounter between the two ships

[0162] Unlike avoiding static obstacles, when avoiding dynamic target ships, the avoidance maneuvers performed by the agent should not only ensure the safety of collision avoidance, but also comply with the relevant requirements of the collision avoidance rules. This section sets up three typical two-ship encounter scenarios, including "head-on encounter", "overtaking" and "cross encounter". The agent and the target ship start from their respective starting positions. When the collision avoidance is completed, the test mission will also terminate at the same time. According to navigation experience, the minimum safe distance that needs to be maintained between the two ships is set to 1 nautical mile. The simulation results of different encounter scenarios are shown in Figure 2. Fig.14 As shown, 1 means meeting head-on, 2 means overtaking, and 3 means crossing.

[0163] In the "meeting" scenario, a target ship approaches the agent in the opposite direction, and the target ship has the same speed as the agent. According to the collision avoidance rules, in the "meeting" scenario, both ships are give-way ships. Therefore, when the risk of collision is detected, the two ships choose to turn right at the same time, each passing from the other's port side. When the risk of collision disappears, the two ships eventually return to their original courses. In the experimental results, both ships made large maneuvers to ensure that their intentions can be clearly detected by the other ship, which meets the requirements of the collision avoidance rules for the give-way ship's avoidance action. In addition, a sufficient safe distance was maintained between the two ships throughout the avoidance process.

[0164] In the "overtaking" scenario, both the target ship and the agent are heading due north, and the agent overtakes the target ship from behind. According to the collision avoidance rules, in the "overtaking" scenario, the ship overtaking from behind should actively perform collision avoidance actions and choose to complete the overtaking from the left or right side of the target ship. In this simulation experiment, the agent chose to overtake from the right side of the target ship, and maintained a predefined safety distance of more than 1 mile throughout the avoidance process. Due to the low relative speed of the two ships during the overtaking process, the execution time of the entire collision avoidance process is relatively long, but in the end the agent will complete the overtaking and return to the initial heading. At the same time, the steering process of the agent changes smoothly, without abrupt changes in heading, which is consistent with the dynamic characteristics of the ship.

[0165] In the "crossing encounter" scenario, the target ship approaches from the starboard side of the smart ship at a 45° angle, and its speed is the same as that of the intelligent agent. According to the collision avoidance rules, in the "crossing encounter" scenario, the ship on the left should give way to the ship on the right, and if possible, the ship giving way should avoid crossing the bow of the straight-line ship. Therefore, in this experimental scenario, the intelligent agent chose to turn right and pass by the stern of the approaching ship to give way. During the entire avoidance process, the intelligent agent and the approaching ship always maintained a sufficient safe distance. When the intelligent agent and the approaching ship completed the avoidance task, they returned to their original course.

[0166] (3) Multiple ship encounter scenarios

[0167] In this experimental scenario, a more complex experimental scenario involving three ships is set up, and the agent needs to avoid two approaching ships under the premise of considering collision avoidance rules and good seamanship. The initial position and heading of each ship in the experimental scenario are pre-set. If the agent does not take any avoidance measures, then a collision will be inevitable. The minimum safe distance between the agent and the approaching ship is still taken as the empirical value of 1 nautical mile.

[0168] Fig.15 The simulation results of the multi-ship collision avoidance scenario are shown in Figure 1, where 1 represents the rudder angle, 2 represents the heading, and 3 represents the distance. In this experimental scenario, the two target ships are defined on the starboard side of the agent. For multi-ship encounter scenarios, the collision avoidance rules do not set specific avoidance action constraints, but only require ships to consider good seamanship when performing avoidance. Fig.15In the trajectory shown in the figure, at the initial moment, the agent chooses to maintain the original course, but as the distance from target ship 1 decreases, the agent performs a rudder angle to the starboard. Because the agent detects the risk of collision with target ship 2, it does not immediately return to the original course, but chooses to continue sailing on the new course. When the risk of collision completely disappears, the agent chooses a left turn and returns to the original course. In the process of collision avoidance, the amplitudes of the two collision avoidance actions performed by the agent are relatively moderate, and the extreme operation of full rudder is not adopted due to the complexity of the scene. During the entire avoidance process, the maximum course change of the agent is 73°, which allows the target ship to clearly identify the avoidance intention of the agent. At the same time, the navigation trajectory of the agent is very smooth, because the maneuverability characteristics of the ship are fully considered in the algorithm model. In addition, the agent chooses to turn to the right and pass the target ship on the port side, which also meets the relevant requirements of good seamanship. The minimum distances between the agent and the two target ships are 3216 meters and 3878 meters respectively, both of which can meet the safety requirements.

[0169] (4) Scenarios where direct-line vessels need to take action

[0170] When unmanned ships are sailing at sea, they will be affected by a lot of environmental uncertainties. Sometimes, due to some uncontrollable reasons, the give-way ship cannot take evasive action immediately. As a result, the distance between the two ships becomes smaller and smaller, and the risk of collision becomes greater and greater. In this case, the collision avoidance rules require that the straight-line ship should give up the right to go straight and take the most helpful action to avoid collision. In order to verify the effectiveness of the collision avoidance framework for this scenario, the present invention sets up a Fig.16 The intersection scenario shown, where 1 represents the rudder angle, 2 represents the heading, and 3 represents the distance.

[0171] Fig.16The collision avoidance simulation results of this experimental scenario are shown in Figure 2. At the initial moment, there is a target ship approaching on the port side of the agent. According to the collision avoidance rules, the target ship on the left side is the give-way ship, and the agent needs to maintain the initial heading and speed. However, due to some unknown reasons, as the distance between the two ships continues to approach, the target ship obviously does not take timely collision avoidance actions in accordance with the requirements of the collision avoidance rules. Therefore, the agent decides to avoid the collision through its own maneuvers. In the simulation experiment, the agent first performed a large right turn, which quickly alleviated the risk of collision between the two ships. At the same time, this large-scale action also enabled the target ship to clearly identify the collision avoidance intention of the agent. In addition, according to the provisions of the collision avoidance rules, in this cross-encounter scenario, if the situation permits, the straight-line ship should avoid taking an evasive action of turning left to the ship on the port side when performing the avoidance action. From the simulation experiment results, it can be found that the agent fully considers the relevant requirements of the collision avoidance rules. In addition, the heading of the agent changes very smoothly, without large fluctuations, and finally returns to the original heading. The minimum distance between the agent and the target ship is about 1949m, which is greater than the specified threshold.

[0172] The present invention proposes a collision avoidance method based on deep reinforcement learning. In order to consider the maneuvering characteristics of the ship, the MMG model is introduced, through which the real-time navigation information of the ship can be inferred. Then, a state space and action space definition method considering the scene and driving experience is designed, and a new collision avoidance decision network construction and training framework is proposed. In addition, in order to fully consider the rules, a set of hierarchical reward function combinations are designed and used for the training of the later decision network. Subsequently, by introducing the DQN algorithm, a collision avoidance decision network training model is constructed, and after a large amount of training, a collision avoidance decision network applicable to different scenarios is obtained. Finally, in order to verify the applicability of the proposed method, a variety of simulation test scenarios are designed, and a comprehensive performance evaluation is carried out. The simulation results show that this method can enable ships to avoid collisions safely in complex environments, while ensuring that they meet the requirements of collision avoidance rules and good seamanship. This method provides a new solution to the collision avoidance problem of unmanned ships.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints, characterized in that: include: Identify the encounter scenario; according to different encounter scenarios, define three state spaces, namely the state spaces in multi-ship encounter, two-ship encounter and static obstacle avoidance scenarios; among them, the state space in the multi-ship encounter scenario consists of 4 parts and contains 22 elements, the state space in the two-ship encounter scenario consists of 3 parts and contains 8 elements, and the state space in the static obstacle avoidance scenario consists of 3 parts and contains 6 elements; For the identified encounter scenario, a decision network corresponding to the encounter scenario is used to execute a ship collision avoidance decision, wherein the decision network corresponding to the encounter scenario is trained by a reward function corresponding to the encounter scenario, and the reward function is set in combination with the collision avoidance rule for the encounter scenario; the reward function is set in layers, wherein the first layer is a basic layer, and the reward function defined in the basic layer takes into account the safety of the ship during the collision avoidance process, general ship handling experience and good seamanship, and all ship encounter scenarios need to execute the reward function in the basic layer; the second layer is a rule layer, and the reward function defined in the rule layer is based on the avoidance requirements of the collision avoidance rules for different encounter scenarios. The reward functions in the rule layer do not need to be executed by all ships, but only need to select the corresponding reward functions according to the actual encounter scenario; the reward function value finally obtained by the ship is the sum of the reward functions of the basic layer and the rule layer; the reward functions defined in the basic layer include the target reward function, the forward reward function, the collision reward function, the rudder angle reward function and the yaw reward function; the reward functions defined in the rule layer include the straight-line ship reward function, the give-way ship reward function, the encounter scenario reward function, the cross encounter scenario reward function and the overtaking scenario reward function; based on the combination of hierarchical reward functions, the DQN algorithm is introduced to train the decision network; Output the collision avoidance action that should be performed.

2. According to the ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints as described in claim 1, it is characterized in that: According to the action constraints of collision avoidance rules for different encounter scenarios, all encounter scenarios are divided into N types, and a corresponding reward function is set for each encounter type to form N reward functions; wherein each reward function is used to train a decision network.

3. According to the ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints as described in claim 1, it is characterized in that: The method further comprises: constructing a ship motion model for establishing a conversion relationship between a current state of the ship and a state at a next moment.

4. According to the ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints as described in claim 1, it is characterized in that: Considering the maneuvering habits of seafarers in actual collision avoidance, a discrete action space is constructed based on the rudder angle. The action space is a vector ranging from -35° to 35° and containing 11 elements, each of which represents an alternative rudder angle for the ship in the process of avoiding obstacles.

5. The ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints according to claim 1 or 4, characterized in that: After the ship identifies the encounter scene from the environment, the spatial state observed by the ship is used as the input of the decision network corresponding to the encounter scene, and the future reward corresponding to each action in the action space is used as the output of the decision network; then the ship selects the action with the highest future reward to execute.