An unmanned ship collision avoidance decision-making method and system integrating ship length experience

Abstract

The application provides an unmanned ship collision avoidance decision-making method and system fusing captain experience, obtains captain experience by statistically understanding the quantitative understanding of COLREGs of captains driving ships of different lengths, and improves a reward function and an action space of a proximal policy optimization algorithm by using the captain experience. The application can adapt to various encounter scenes and solve the problem of insufficient consideration of COLREGs by developing a new reward function based on captain experience. The unmanned ship can improve the intelligent collision avoidance decision-making ability of the unmanned ship under the scheme provided by the application, and fully meet the requirements of COLREGs.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned ship control and collision avoidance, and particularly relates to an unmanned ship collision avoidance decision-making method and system fusing ship captain experience. BACKGROUND

[0002] An unmanned ship is a self-driving watercraft, which is generally composed of a power control system, an intelligent collision avoidance decision-making system, a situation awareness system, a ship-shore communication system and the like, and is widely used in military fields (such as patrol, search and rescue operations, ocean exploration, mine clearance, anti-submarine warfare, anti-terrorism operations and pirate suppression tasks) and civil fields (such as cargo transportation, channel measurement and maritime patrol tasks). In actual navigation, many static obstacles and dynamic ships will be encountered, and in order to successfully complete the task, the unmanned ship needs to take reasonable and efficient avoidance measures for the encountered obstacles to ensure navigation safety.

[0003] In the research of unmanned ships, there are generally three types of algorithms for collision avoidance problems: 1) control-based collision avoidance algorithm: using a ship motion mathematical model to predict the behavior of the ship, and making collision avoidance decisions according to the prediction results, this algorithm is mainly based on model predictive control algorithm, which often requires a large amount of calculation cost and is difficult to meet the real-time requirements of ships. 2) optimization-based collision avoidance algorithm: by considering multiple factors such as ship speed, distance, direction and environmental conditions, the best collision avoidance strategy is found, this algorithm is mainly based on artificial potential field method, genetic algorithm and the like, which has the problems of ignoring the dynamics of the ship and easily falling into local optimal solution. 3) reinforcement learning-based collision avoidance algorithm: allows the ship to learn the best behavior strategy through interaction with the environment and combining the dynamics of the ship, which can maximize the safety and adaptability of collision avoidance decisions, but the algorithm takes avoidance measures that do not conform to the avoidance habits of most captains and violate the International Regulations for Preventing Collisions at Sea (COLREGs) in many encounter situations.

[0004] Although great progress has been made in recent years in the research of ship intelligent collision avoidance, the above algorithms can all solve the problem of intelligent collision avoidance of unmanned ships facing static obstacles and dynamic ships to some extent, but in the special case where the giving-way ship does not take avoidance measures, it is difficult to give reasonable avoidance schemes for the oncoming ship in time to ensure navigation safety.

[0005] After searching, it is found that the closest existing designs generally include the following documents:

[0006] The patent application number CN113687658A publishes a model predictive control method for multi-ship real-time automatic collision avoidance in the patent "Model predictive control method for multi-ship real-time automatic collision avoidance";

[0007] The patent application number CN115167404A publishes a kind of sea autonomous water surface ship collision avoidance decision-making method based on transfer reinforcement learning in the patent "a kind of sea autonomous water surface ship collision avoidance decision-making method based on transfer reinforcement learning".

[0008] The patent application number CN114895673A publishes a kind of ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints in the patent "a kind of ship collision avoidance decision-making method based on deep reinforcement learning under rule constraints".

[0009] The patent application number CN116540698A publishes a kind of ship collision avoidance path planning method based on improved artificial potential field model in the patent "a kind of ship collision avoidance path planning method based on improved artificial potential field model".

[0010] However, the above-mentioned prior art solutions cannot fully comply with COLREGs, and do not consider the special case where the straight-ahead ship does not take effective avoidance measures. The model predictive control algorithm requires a large amount of calculation in the avoidance decision-making process, making it difficult to meet the real-time requirements of the ship algorithm. The artificial potential field method does not adequately consider the mechanical property constraints of the ship's own control system in the avoidance process. Deep reinforcement learning can consider the kinematic properties of the ship and provide a safe solution to the collision avoidance problem in complex marine environments, but it generally only solves the compliance problem of the avoidance decision-making of the ship as the giving-way ship, and has a low degree of compliance with the collision avoidance rules, making it difficult to ensure safety in the case where the ship is a straight-ahead ship but the other ship does not take effective avoidance measures. SUMMARY

[0011] To solve the problem that the existing solutions do not fully comply with the collision avoidance method of COLREGs, and to achieve safe navigation of unmanned ships in straight-ahead and giving-way situations, it is necessary to study an unmanned ship collision avoidance decision-making algorithm that complies with the habits of ship captains, and to achieve safe navigation of unmanned ships at sea, the present invention proposes an unmanned ship collision avoidance decision-making scheme based on the experience of ship captains and proximal policy optimization, based on a deep reinforcement learning algorithm that integrates the experience of ship captains, to achieve decision-making for unmanned ship collision avoidance.

[0012] In this scheme, a questionnaire is designed to statistically understand the quantitative understanding of COLREGs by ship captains driving ships of different lengths, to obtain the experience of ship captains, and to improve the reward function and action space of the proximal policy optimization algorithm. By developing a new reward function based on the experience of ship captains, the present invention can adapt to various encounter scenarios and solve the problem of insufficient consideration of COLREGs. The unmanned ship can improve its intelligent collision avoidance decision-making ability and fully comply with the requirements of COLREGs.

[0013] The present invention specifically adopts the following technical solutions:

[0014] An unmanned ship collision avoidance decision-making method fusing captain experience, which obtains captain experience by statistically quantifying the understanding of COLREGs of captains driving ships of different lengths, obtains the relationship function between ship length and the distance of the captain taking avoidance measures to meet COLREGs, the amplitude of the change in the heading, the safe meeting distance, and the distance at which the straight-ahead ship can take avoidance measures when the giving-way ship does not take avoidance measures by fitting the data, improves the reward function and action space of the proximal policy optimization algorithm and trains it to output unmanned ship collision avoidance decisions.

[0015] Further, the captain experience includes: statistically obtaining the ship distance that meets the requirement of "taking avoidance measures as early as possible" in COLREGs, the amplitude of the change in the heading that meets the requirement of "sufficient observation of the change in the heading / speed", and the distance between the two ships when the straight-ahead ship needs to take avoidance measures if the giving-way ship has not taken effective avoidance measures.

[0016] The three parameters are fitted to obtain the functional relationship between the ship length.

[0017] Further, the specific process of obtaining the captain experience data is as follows:

[0018] The distance P1(x) and the amplitude of the change in the heading P2(x) information of the captain driving a ship of different lengths when the ship is the giving-way ship and the distance information P3(x) of the captain considering taking avoidance measures when the ship is the straight-ahead ship are respectively statistically obtained; the least squares method is used to fit the data of the captain driving the ship, and the fitting formula is as follows:

[0019]

[0020] where x is the captain, a0, a1, a2, …, a n , b0, b1, b2, …, b n , c0, c1, c2, …, c n are polynomial coefficients.

[0021] The requirements of "early, large, wide, and clear" in the rules are considered in the avoidance process, and the reward function that meets the captain experience is set according to the different avoidance responsibilities of the unmanned ship as follows:

[0022] 1) When the ship is the giving-way ship:

[0023] R cg = α1R cg1 + α2R cg2

[0024]

[0025] wherein p1(x), p2(x) represent the distance and the change of heading amplitude of the unmanned ship when taking avoidance measures, and a1, a2 are the preference weights of the ship length for the "early" and "large" in the rules, obtained by the questionnaire and a1+a2=1; to ensure that the unmanned ship can obtain greater rewards when taking larger amplitude avoidance measures as early as possible;

[0026] 2) When the ship is a straight-ahead ship, the behavior of the ship is divided into three categories: one, when the distance between the two ships is too large, the ship should take avoidance measures to keep speed and direction; two, when the distance between the two ships is less than or equal to the distance at which the straight-ahead ship can take avoidance measures, if the other ship takes effective avoidance measures, the ship should still take avoidance measures to keep speed and direction; three, when the distance between the two ships is less than or equal to the distance at which the straight-ahead ship can take avoidance measures, if the other ship does not take effective avoidance measures, the ship should take avoidance measures to ensure the safety of the heading; for the three cases, the reward functions are given as follows:

[0027] a) When the distance between the two ships is greater than P3(x):

[0028]

[0029] wherein Δψ is the change of the ship's heading, and Δv is the change of the ship's speed;

[0030] b) When the distance between the two ships is less than or equal to P3(x) and the other ship takes effective measures:

[0031]

[0032] c) When the distance between the two ships is less than or equal to P3(x) but the other ship does not take effective measures:

[0033]

[0034] Further, the training process of the near-end strategy optimization algorithm is described by the distribution functions p, q, which includes the following steps:

[0035] Step 3.1, determine the position, navigation situation information of the ship and the other ship, and the target point of the ship through the designed unknown environment;

[0036] Step 3.2, calculate the closest encounter distance and the closest encounter time between the two ships, evaluate the current collision risk, and feed back the results to the near-end strategy optimization network π to perform actions and update the position state and action state, and obtain the network parameters θ';

[0037] Step 3.3, the network π old make decisions according to the environment to obtain the network parameters θ;

[0038] Step 3.4, θ and θ' are updated by KL divergence θ' to θ;

[0039] Step 3.5, if the collision risk is detected, change the unmanned ship heading to avoid obstacles;

[0040] Step 3.6, if the distance to the target point is getting farther and farther, feedback a lower reward value, and adjust the motion direction of the unmanned ship to make it constantly close to the target point;

[0041] Step 3.7, judge whether the collision avoidance is completed, if not, return to step 3.1;

[0042] Step 3.8, judge whether the target point is reached, if the set target point is reached, the basic collision avoidance route is obtained, otherwise continue to iterate and update until the unmanned ship reaches the target point;

[0043] Step 3.9, train several times to generate the optimal collision avoidance route algorithm, and the training model is obtained.

[0044] Further, the unmanned ship is a single-screw single-rudder under-actuated unmanned ship, the maneuvering condition of the unmanned ship is described according to the separated structure model, and the model is as follows:

[0045]

[0046] Where (x, y, ψ) represents the position and heading of the unmanned ship in the inertial coordinate system; (u, v, r) represents the linear velocity and turning velocity of the unmanned ship in the body coordinate system; m x is the added mass of the unmanned ship in the x-axis; y is the added mass of the unmanned ship in the y-axis; ZZ is the moment of inertia of the unmanned ship around the z-axis; ZZ is the inertia moment of the added mass of the unmanned ship; H , Y H , N H represent the force and moment acting on the ship body; P , Y P , N P represent the force and moment acting on the propeller; R , Y R , N R represent the force and moment acting on the rudder; E , Y E , N E represent the force and moment of the external environment disturbance acting on the ship; x c represents the coordinate of the ship center in the x-axis of the body coordinate system; c is the command rudder angle; δ is the actual rudder angle; T E is the rudder time constant; n c is the command propeller speed; n is the actual propeller speed; T Mare the propeller characteristic constants.

[0047] Further, on the basis of training the model, based on the test environment and the current unmanned ship basic information, the model is tested through various encounter scenes to verify the effectiveness of the model; wherein the surrounding environment information is uniformly represented by a vector, as the model input for decision-making;

[0048] In view of the problem that the surrounding environment of the unmanned ship will change during navigation, the model input dimension will also change, GRU neural network is adopted to process the input information into the same dimension, and the data information of each ship is extracted and retained.

[0049] Further, in the test process, the sensor of the unmanned ship sets the maximum detection range, and when the other ship exists within the detection range, it is judged whether to take avoidance measures; when the collision risk of the two ships exceeds the threshold, the collision avoidance decision is calculated and executed; and according to the distance between the current state of the unmanned ship and the target point, the reward is fed back to guide the decision-making behavior of the unmanned ship at the next moment; the model learns the action strategy by constantly interacting with the environment, and the learning effect is represented by the cumulative reward value of each training event, and the total reward value is calculated; according to the test result and the feedback reward value, it is judged whether the unmanned ship reaches the target point.

[0050] Further, according to the navigation state of the two ships, the closest encounter distance DCPA and the closest encounter time TCPA are calculated, and the fuzzy comprehensive evaluation is combined to calculate the ship collision risk CRI, when CRI>0.5, the model makes the giving-way ship take collision avoidance decision in time; the ship collision risk calculation formula is as follows:

[0051] CRI=β1R DCPA +β2P TCPA

[0052]

[0053] Wherein D is the DCPA between the two ships; T is the TCPA between the two ships; d1 is the safe encounter distance of the ship; d2 is the absolute safe encounter distance of the ship; V is the relative motion speed of the two ships; R DCPA , R TCPA are the membership functions of DCPA and TCPA respectively; β1, β2 are the weights of R DCPA , R TCPA .

[0054] Further, a reward evaluation function is set for each step of action, a positive reward is given for approaching the target point and avoiding obstacles, and vice versa, so that the unmanned ship reaches the target point in the shortest time and optimal path; wherein the reward function is set as follows:

[0055]

[0056] R c = (1 - mu) R cs + mu R cg

[0057] wherein p1, p2, p3, p4, p5, p6 are constant parameters, set according to the environment, for adjusting the reward function; d represents the change of the distance between the unmanned ship and the destination in the t time interval, v represents the speed of the unmanned ship, R dd is the distance reward function, set in the range of (0, 1), which is proportional to the distance change, that is, the closer the unmanned ship is to the target point, the greater the reward value; R t is the time reward function, set in the range of (0, 1), which is inversely proportional to the time, that is, the shorter the time, the greater the reward value; t min is the predicted shortest time for the unmanned ship to collide with the obstacle at the current speed; R c is the captain experience reward function, set in the range of (0, 1), the greater the reward is, the more in line with the captain's experience, and mu is a Boolean variable, mu = 1 when the ship is a yielding ship, otherwise mu = 0.

[0058] And an unmanned ship collision avoidance decision system fusing captain experience, comprising a memory, a processor and computer program instructions stored on the memory and capable of being executed by the processor, when the processor executes the computer program instructions, the algorithm model obtained and trained by the decision method described above is outputted to output the unmanned ship collision avoidance decision.

[0059] Compared with the prior art, the collision avoidance algorithm provided by the present application and the preferred scheme thereof is based on the proximal policy optimization algorithm, improves the reward function for collision avoidance decision by fusing the captain's experience, evaluates the collision risk degree of the ship according to the position and sailing situation information of the ship and the opposite ship, and calculates the collision avoidance decision in line with COLREGs according to the risk degree, thereby ensuring the safe sailing of the unmanned ship.

[0060] The algorithm fuses the proximal policy optimization and the captain's experience, can ensure that the distance and the avoidance range of the avoidance measure taken by the ship when the ship is a yielding ship are more in line with the requirements of COLREGs, can ensure that the yielding ship can take avoidance measures in time when the ship is a straight sailing ship and faces the situation that the yielding ship does not take effective avoidance measures, ensures the safety of sailing, improves the collision avoidance efficiency of the algorithm for special situations, and has good generalization ability, thereby ensuring the safety of the unmanned ship sailing at sea. BRIEF DESCRIPTION OF DRAWINGS

[0061] The present application will be further described in detail below in combination with the drawings and specific embodiments:

[0062] Figure 1 It is the flow chart of the unmanned ship collision avoidance decision of the embodiment of the present application.

[0063] Figure 2 The structure diagram of the proximal strategy optimization algorithm for the embodiment of the application.

[0064] Figure 3 The flow chart of data processing for the embodiment of the application.

[0065] Figure 4 The structure schematic diagram of the single oar and single rudder unmanned ship for the embodiment of the application. DETAILED DESCRIPTION

[0066] In order to make the features and advantages of the patent more obvious and easy to understand, the following embodiments are specifically described as follows:

[0067] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the specification have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0068] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0069] The embodiment of the application proposes an unmanned ship collision avoidance decision scheme based on the proximal strategy optimization algorithm, which obtains the relationship function between the distance of the captain's taking avoidance measures to meet the COLREGs, the amplitude of the heading change, the safe meeting distance, and the relationship between the distance of the straight-ahead ship taking avoidance measures and the ship length when the giving-way ship does not take avoidance measures by designing a questionnaire to count the captain's experience, and improves the reward function of the proximal strategy optimization algorithm to solve the problem that the existing collision avoidance algorithm does not fully consider the requirements of COLREGs.

[0070] The unmanned ship collision avoidance decision process refers to Figure 1 and includes the following steps:

[0071] Step 1, count the captain's experience;

[0072] Step 2, build a collision avoidance decision model;

[0073] Step 3, model training;

[0074] Step 4, design a test environment;

[0075] Step 5, environment perception;

[0076] Step 6, data processing;

[0077] Step 7, judge whether there is a risk of collision between ships;

[0078] Step 8, according to step 7, if there is a risk of collision, judge the collision avoidance responsibility of the ship and execute the corresponding collision avoidance decision;

[0079] Step 9, according to step 8, calculate the reward value;

[0080] Step 10, judge whether the destination is reached or the avoidance is completed and return the reward value and the result.

[0081] For step 1, the main statistics of the ship length in open water are the distance of the ship length that meets the requirements of "early avoidance behavior" in COLREGs, the heading change amplitude that meets the requirements of "sufficient observation of heading / speed change", and the distance of the two ships that the straight-ahead ship needs to take avoidance measures to meet the requirements of "if the giving-way ship has not taken effective avoidance measures, the straight-ahead ship can take avoidance measures alone to ensure the safety of navigation".

[0082] For step 1, the specific steps are as follows:

[0083] For step 1.1, design the questionnaire according to the investigation content

[0084] For step 1.2, generate the questionnaire and forward the questionnaire link to the three groups of people, i.e. ship captains, crew members and teachers of maritime colleges, and collect the questionnaire data through the background.

[0085] For step 1.3, through the questionnaire data, the data fitting of the three parameters of "the distance between the two ships when taking avoidance measures", "the heading change amplitude" and "the distance of the ship that the straight-ahead ship can take avoidance measures" is carried out, and the functional relationship between the ship length and the parameters is obtained.

[0086] For step 2, the proximal strategy optimization algorithm is a three-network structure, and the algorithm structure is as shown in Figure 2 The algorithm sets two actor networks, which are two layers with 256 neurons in each layer. Among them, the network π samples, and the old network π old is updated. During the training cycle, π receives the current environment information, selects the action according to the information, updates the state s and returns the reward r. The two actor networks are updated through adaptive KL penalty, and the critic network structure is two layers with 256 neurons in each layer, which evaluates the action through s and r, and updates π. The fitting function obtained in step 3 is combined with the ship length of the test ship to obtain the parameter value, and the reward function is designed to make the avoidance behavior consistent with the captain's experience to obtain higher reward, and to improve the compliance of the collision avoidance decision model to the collision avoidance rules.

[0087] For step 3, the goal of the proximal policy optimization algorithm is to maximize the expected reward, which is calculated as follows:

[0088]

[0089] The distribution functions p, q describe the two boats. q can be any distribution, and p is close to q in the algorithm, and the variance of the distribution is calculated as follows:

[0090] Var x～p [f(x)]=E x～p [f(x) 2 ]-(E x～p [f(x)]) 2 (2)

[0091]

[0092] When the sampling data is sufficient, p(x) = q(x).

[0093] Using the importance sampling method, the online policy is converted to an offline policy. In the policy gradient, the solution to the expectation

[0094]

[0095] is converted to:

[0096]

[0097] where is the correction term.

[0098] Applied to the actual environment, the gradient is updated

[0099]

[0100]

[0101] where A θ (s t ,a t ) is the evaluation function, which evaluates the goodness of choosing action a in state s at time t.

[0102] The new optimization function

[0103]

[0104] The proximal policy optimization definition is obtained from the above formula

[0105]

[0106] Wherein β is a weight coefficient, the role of KL divergence is to describe the difference between θ, θ' difference measure, difference refers to the difference of the behavior (actor) corresponding to the parameter. βKL(θ, θ') is a constraint condition.

[0107] For step 3, the operation process of algorithm training model is specifically divided into the following steps:

[0108] Step 3.1, through the designed unknown environment, the position of the ship and the other ship, the sailing situation information and the target point of the ship are determined;

[0109] Step 3.2, the closest meeting distance and the closest meeting time between the two ships are calculated, the current collision risk is evaluated, and the results are fed back to the near-end strategy optimization, the network π executes the action and updates the position state and action state, and the network parameter θ' is obtained;

[0110] Step 3.3, network π old According to the environment, the network parameter θ is obtained;

[0111] Step 3.4, θ and θ' are updated by KL divergence θ' to θ;

[0112] Step 3.5, if the collision risk is detected, change the heading of the unmanned ship to avoid the obstacle;

[0113] Step 3.6, if the distance to the target point is getting farther and farther, then feedback a lower reward value, adjust the motion direction of the unmanned ship to make it constantly close to the target point;

[0114] Step 3.7, judge whether the collision avoidance is completed, if not, return to step 3.1;

[0115] Step 3.8, judge whether the target point is reached, if the set target point is reached, the basic collision avoidance route is obtained, otherwise continue to iterate and update until the unmanned ship reaches the target point;

[0116] Step 3.9, train several times to generate the optimal collision avoidance route algorithm training is completed and the training model is obtained.

[0117] For step 4, design test environment, obtain test environment and current unmanned ship basic information.

[0118] For step 5, real-time monitoring of unmanned ship information.

[0119] For step 6, the GRU neural network processes the input information into the same dimension.

[0120] For step 7, the sensor of the unmanned ship needs to set the maximum detection range, and when there is an opposite ship in the detection range, it is judged whether to take evasive measures.

[0121] For step 8, the unmanned ship executes the collision avoidance decision given by the algorithm.

[0122] For step 9, according to the distance between the current state of the unmanned ship and the target point, a reward is fed back to guide the decision-making behavior of the unmanned ship at the next moment.

[0123] For step 10, the model learns the action strategy by constantly interacting with the environment, and the learning effect is represented by the cumulative reward value of each training event, and the total reward value is calculated.

[0124] Further, the following further demonstrates and introduces the content of the application in combination with a specific design case:

[0125] In view of the fact that the existing algorithm does not fully consider the international maritime collision avoidance rules, the embodiment of the application first considers improving the proximal policy optimization algorithm by adding the captain's experience.

[0126] One of the technical problems to be solved for realizing autonomous navigation of the unmanned ship is to avoid collision between ships, and in a complex sea environment, a good collision avoidance decision is needed to ensure safe navigation of the unmanned ship. The proximal policy optimization algorithm is applied to intelligent collision avoidance of the unmanned ship, but the constraints of the international maritime collision avoidance rules need to be considered in the collision avoidance process. Through a questionnaire survey, the captain's experience data is obtained, the reward function is improved, and the problem of intelligent collision avoidance under the constraint of the collision avoidance rules is solved.

[0127] The process of obtaining the captain's experience data by questionnaire survey is as follows:

[0128] The distance P1(x) and the heading change amplitude P2(x) information of the captain driving the ship for the case of the ship being a giving-way ship, and the distance information P3(x) that the ship can consider taking evasive measures for the case of the ship being a straight-ahead ship are respectively counted. The least squares method is used for data fitting combined with the captain driving the ship, and the fitting formula is as follows:

[0129]

[0130] Where x is the captain, a0, a1, a2, …, a n , b0, b1, b2, …, b n , c0, c1, c2, …, c n are polynomial coefficients.

[0131] In actual navigation of the unmanned ship, the encountered ships need to be avoided according to the International Maritime Collision Regulations, and the avoidance process needs to comply with the requirements of "early, large, wide, and clear" in the regulations. In order to make the collision avoidance decision of the unmanned ship more in line with this specification, according to the different avoidance responsibilities of the unmanned ship, the reward function in line with the captain's experience is increased as follows:

[0132] 1) When the ego ship is the giving-way ship:

[0133] R cg = a1R cg1 + a2R cg2 (10)

[0134]

[0135] where p1(x), p2(x) represent the distance and the amplitude of the change in the heading of the ego ship when taking the giving-way measures, and a1, a2 are the weights of the ego ship's preference for the "early" and "large" in the rules, which are obtained by the questionnaire and a1+a2=1. The formula can ensure that the ego ship can obtain greater rewards when taking the giving-way measures early and with a large amplitude.

[0136] 2) When the ego ship is the keeping course ship, the behavior of the ego ship mainly includes three types: 1) when the distance between the two ships is too large, the ego ship should take the giving-way measures to keep the speed and the heading; 2) when the distance between the two ships is less than or equal to the distance at which the keeping course ship can take the giving-way measures, if the other ship takes effective giving-way measures, the ego ship should still take the giving-way measures to keep the speed and the heading; 3) when the distance between the two ships is less than or equal to the distance at which the keeping course ship can take the giving-way measures, if the other ship does not take effective giving-way measures, the ego ship should take the giving-way measures to ensure the safety of the heading. For the three types, the designed reward functions are as follows:

[0137] a) when the distance between the two ships is greater than P3(x):

[0138]

[0139] where Δψ is the change in the heading of the ego ship, and Δv is the change in the speed of the ego ship. The formula makes the ego ship take the giving-way measures to keep the speed and the heading when the distance between the two ships is too large, and the ego ship can obtain greater rewards.

[0140] b) when the distance between the two ships is less than or equal to P3(x) and the other ship takes effective measures:

[0141]

[0142] The formula makes the ego ship take the giving-way measures to keep the speed and the heading, and the ego ship can obtain greater rewards.

[0143] c) when the distance between the two ships is less than or equal to P3(x) but the other ship does not take effective measures:

[0144]

[0145] The formula makes the ego ship take the necessary giving-way measures, and the ego ship can obtain greater rewards.

[0146] The improved algorithm flowchart provided by the embodiment of the present application is as follows:Figure 1 As shown, the calculation steps are as follows:

[0147] 1. Design a questionnaire based on the survey content, and perform data fitting by statistically analyzing the survey data.

[0148] 2. Construct a decision-making model. The neural network structure consists of two layers, with 256 neurons in each layer.

[0149] 3. Model Training: To facilitate model construction, it is assumed that the unmanned vessel is a single-propeller, single-rudder, underactuated unmanned vessel with the following structure: Figure 4 As shown.

[0150] The maneuverability of this type of unmanned vessel is described using the split-structure model (MMG model), as shown below:

[0151]

[0152] Where (x, y, ψ) represents the USV's position and heading in the inertial coordinate system; (u, v, r) represents the USV's linear velocity and turning velocity in the attached coordinate system; m x m is the additional mass of the USV along the x-axis. y I represents the additional mass of USV on the y-axis. ZZ J is the moment of inertia of USV about the z-axis; ZZ Add a mass moment of inertia to the USV; (X) H ,Y H N H (X) represents the forces and moments acting on the hull; P ,Y P N P (X) represents the force and torque acting on the propeller; R ,Y R N R (X) represents the force and torque acting on the rudder; E ,Y E N E ) represents the forces and moments exerted on the ship by external environmental disturbances; x c This represents the x-coordinate of the ship's center in the appendage coordinate system; δ c δ is the command rudder angle; T is the actual rudder angle. E n is the servo time constant; c T is the commanded rotational speed of the propeller; n is the actual rotational speed of the propeller; M is the propeller characteristic constant.

[0153] 4. Design various encounter scenarios to test the model and verify its effectiveness.

[0154] 5. Environmental information is uniformly represented by vectors and used as input for model decision-making.

[0155] 6, Since the environment around the unmanned ship will change during navigation, the model input dimension will also change. For this case, the GRU algorithm is used to process and extract effective information, and the GRU algorithm process is as shown in Figure 3 n The observation value of the surrounding ship in the detection range is O self , and the state of the ship itself is connected with the state value of the ship itself to form a fixed-length observation value O. This algorithm can retain the data information of each ship, and the normalized observation data can speed up the training process.

[0156] 7, According to the navigation state of the two ships, the closest approach distance DCPA and the closest approach time TCPA are calculated, and the fuzzy comprehensive evaluation is combined to calculate the ship collision risk CRI. When CRI>0.5, the model will make the giving-way ship take collision avoidance decision in time. The ship collision risk calculation formula is as follows:

[0157] CRI=β1R DCPA +β2R TCPA (17)

[0158]

[0159] Where D is the DCPA between the two ships; T is the TCPA between the two ships; d1 is the safe meeting distance of the ship; d2 is the absolute safe meeting distance of the ship; V is the relative motion speed of the two ships; R DCPA , R TCPA are the membership functions of DCPA and TCPA respectively; β1, β2 are the weights of R DCPA , R TCPA .

[0160] 8, When the collision risk of the two ships exceeds the threshold value, the collision avoidance decision is calculated and executed;

[0161] 9, A reward evaluation function is set for each step of action, and a positive reward is given for approaching the target point and avoiding obstacles, and vice versa. Let it reach the target point with the shortest time optimal path. The reward function set by the invention is as follows:

[0162]

[0163] R c =(1-μ)R cs +μR cg (25)

[0164] ​wherein p1, p2, p3, p4, p5, p6 are constant parameters, which are set according to the environment in the experiment, used to adjust the reward function, and then improve the performance of the policy function.d represents the change of the distance between the unmanned ship and the destination in the t time interval, v represents the speed of the unmanned ship, R dd is the distance reward function, which is set in the range of (0, 1), and is proportional to the distance change, that is, the closer the unmanned ship is to the target point, the greater the reward value is. t is the time reward function, which is set in the range of (0, 1), and is inversely proportional to the time, that is, the shorter the time is, the greater the reward value is.t min is the predicted shortest time for the unmanned ship to collide with the obstacle at the current speed.R c is the captain experience reward function, which is set in the range of (0, 1), and the greater the reward is, the more the captain experience is, and μ is a Boolean variable, μ = 1 when the ship is a yielding ship, otherwise μ = 0.

[0165] 10, the test is ended, and whether the unmanned ship reaches the target point is judged according to the test result and the feedback reward value.

[0166] It can be seen from the above that the embodiment of the application combines the proximal policy optimization and the captain experience, combines the advantages of both, so that the unmanned ship can take a more international maritime collision avoidance rule-compliant avoidance decision, and the safe navigation of the unmanned ship is ensured.

[0167] In the embodiment of the application, the algorithm describes the unmanned ship maneuvering situation by using the MMG model, is more practical, and can accurately realize the unmanned ship collision avoidance.

[0168] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system, or a computer program product. Therefore, the application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0169] The application is described with reference to the flowcharts and / or block diagrams according to the method, device (system), and computer program product of the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a machine that implements the functions described in the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocks Figure 1means for performing the function specified by the block or blocks.

[0170] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a Figure 1 one or more flowcharts and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0172] Each technical feature in the above-described embodiments can be combined in any manner, and to make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, however, as long as the combinations of the technical features do not exist in contradiction, they shall be considered within the scope of the present disclosure.

[0173] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application rather than limiting them, although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that: the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the protection scope of the claims of the present application.

[0174] The patent is not limited to the above best mode, anyone can draw other various forms of a fusion of ship length experience in the patent under the inspiration of unmanned ship collision avoidance decision method and system, any changes and modifications made in accordance with the scope of the patent application of the present application, should be covered within the scope of the patent.

Claims

1. A collision avoidance decision-making method for unmanned ships that integrates the experience of human pilots, characterized in that: The quantitative understanding of COLREGs by the captain of the ship driving the ship of different lengths is obtained, the captain's experience is obtained, the relationship function between the distance of the captain to take avoidance measures to meet the COLREGs, the amplitude of the heading change, the safe meeting distance, and the distance of the straight-ahead ship to take avoidance measures when the giving-way ship does not take avoidance measures and the length of the ship is obtained by fitting the data, the reward function and the action space of the proximal policy optimization algorithm are improved and trained to output the unmanned ship collision avoidance decision; The closest point of approach distance DCPA and the closest point of approach time TCPA are calculated according to the sailing state of the two ships, and the ship collision risk CRI is calculated by combining the fuzzy comprehensive evaluation, and when CRI>0.5, the giving-way ship takes timely collision avoidance decision through the model; the ship collision risk calculation formula is as follows: CRI = β1R DCPA + β2R TCPA where D is the DCPA between two ships; T is the TCPA between two ships; d1 is the safe passing distance of a ship; d2 is the absolute safe passing distance of a ship; V is the relative motion speed of two ships; R DCPA , R TCPA are the membership functions of DCPA and TCPA respectively; β1, β2 are the weights of R DCPA , R TCPA . A reward evaluation function is set for each step action, a positive reward is given for approaching the target point and avoiding obstacles, and a negative reward is given otherwise, so that the ship reaches the target point in the shortest time and the optimal path; wherein the reward function is set as follows: R c = (1 - μ)R cs + μR cg where p1, p2, p3, p4, p5, p6 are constant parameters, set according to the environment, for adjusting the reward function; Δd represents the change of the distance between the unmanned ship and the destination in the t time interval, v represents the speed of the unmanned ship, R d is the distance reward function, set in the range of (0, 1), which is proportional to the distance change, that is, the closer the unmanned ship is to the target point, the greater the distance reward value; R t is the time reward function, set in the range of (0, 1), which is inversely proportional to the time, that is, the shorter the time, the greater the reward value; t min is the predicted shortest time for the unmanned ship to collide with the obstacle at the current speed; R c is the captain experience reward function, set to range in (0, 1), the more the captain experience reward is, the greater the range is, μ is a Boolean variable, μ = 1 when the ship is a yielding ship, otherwise μ = 0, R cg is the reward function of the captain experience when the ship is a yielding ship.

2. The unmanned ship collision avoidance decision-making method fusing the captain's experience according to claim 1, characterized in that: The captain's experience includes: statistics of the captain's understanding of the ship distance that meets the "early avoidance behavior" requirement in COLREGs, the amplitude of the heading change that meets the "sufficient observation of the heading / speed change" requirement, and the distance between the two ships when the giving-way ship needs to take avoidance measures; The three parameters are fitted to obtain the functional relationship between the ship length.

3. The unmanned ship collision avoidance decision-making method fusing the captain's experience according to claim 2, characterized in that: The specific process of obtaining the captain's experience data is as follows: The distance P1(x) and the amplitude of the heading change P2(x) information of the captain driving the ship of different lengths to take avoidance measures when the ship is the giving-way ship, and the distance information P3(x) of the captain considering taking avoidance measures when the ship is the straight-ahead ship are respectively counted; the least squares method is used to fit the data according to the captain driving the ship, and the fitting formula is as follows: where x is the ship length, a0, a1, a2,..., an are polynomial coefficients; n b0, b1, b2,..., bn are polynomial coefficients; n c0, c1, c2,..., cn are polynomial coefficients; n are polynomial coefficients; Considering the requirements of "early, large, wide, and clear" in the rules during the avoidance process, the reward function consistent with the captain's experience is set according to the different avoidance responsibilities of the unmanned ship as follows: 1) When the ship is the giving-way ship: R cg = a1R cg1 + a2R cg2 Wherein p1(x) and p2(x) represent the distance and the amplitude of the heading change when the unmanned ship takes avoidance measures, and α1 and α2 are the preference weight of the captain for the "early" and "large" in the rules, which are obtained by the questionnaire and α1+α2=1; to ensure that the unmanned ship can obtain more rewards when taking early and large avoidance measures; 2) When the ship is a straight sailing ship, the behavior of the ship is divided into three types: 1) when the distance between the two ships is too large, the ship should take measures to avoid the speed and direction; 2) when the distance between the two ships is less than or equal to the distance that the straight sailing ship can take to avoid the measures, if the other ship takes effective avoidance measures, the ship should still take measures to avoid the speed and direction; 3) when the distance between the two ships is less than or equal to the distance that the straight sailing ship can take to avoid the measures, if the other ship does not take effective avoidance measures, the ship should take measures to ensure the safety of the heading; for the three cases, the reward functions are given as follows: a) when the distance between the two ships is greater than P3(x): where Δψ is the change of the ship's heading, and Δv is the change of the ship's speed; b) when the distance between the two ships is less than or equal to P3(x) and the other ship takes effective measures: c) when the distance between the two ships is less than or equal to P3(x) but the other ship does not take effective measures:

4. The unmanned ship collision avoidance decision-making method according to claim 1, characterized in that: The distribution functions p and q are used to describe two unmanned ships, and the training process of the proximal policy optimization algorithm includes the following steps: Step 3.1, determine the position, sailing situation information of the ship and the other ship, and the target point of the ship through the designed unknown environment; Step 3.2, calculate the closest approach distance and time between the two ships, evaluate the current collision risk, and feed back the results to the proximal policy optimization network π to perform actions and update the position state and action state to obtain the network parameter θ'; Step 3.3, network π old Decide according to environment, get network parameter θ; Step 3.4, θ and θ' are updated by KL divergence θ' to θ; Step 3.5, if the collision risk is detected, change the heading of the unmanned ship to avoid obstacles; Step 3.6, if the distance to the target point is getting farther and farther, feedback a lower reward value, and adjust the motion direction of the unmanned ship to make it constantly approach the target point; Step 3.7, judge whether the collision avoidance is completed, if not, return to step 3.1; Step 3.8, judge whether the target point is reached, if the set target point is reached, the basic collision avoidance route is obtained, otherwise continue to iterate and update until the unmanned ship reaches the target point; Step 3.9, train several times to generate the optimal collision avoidance route algorithm training is completed and the training model is obtained.

5. The unmanned ship collision avoidance decision-making method according to claim 4, characterized in that: The unmanned ship is a single propeller and single rudder underactuated unmanned ship, and the maneuvering of the unmanned ship is described according to the separation type structure model, as shown below: where (x, y, ψ) represents the position and heading of the USV in the body-fixed frame; (u, v, r) represents the linear and rotational velocities of the USV in the body-fixed frame; m x is the added mass of the USV in the x-axis; m y is the added mass of the USV in the y-axis; I ZZ is the moment of inertia of the USV about the z-axis; J ZZ is the inertia tensor of the USV; (X H , Y H , N H ) represents the forces and moments acting on the hull; (X P , Y P , N P ) represents the forces and moments acting on the propeller; (X R , Y R , N R ) represents the forces and moments acting on the rudder; (X E , Y E , N E ) represents the external environmental disturbances acting on the vessel; x c represents the x-coordinate of the vessel center in the body-fixed frame; δ c is the commanded rudder angle; δ is the actual rudder angle; T E is the time constant of the rudder; n c is the commanded propeller speed; n is the actual propeller speed; T M is the propeller characteristic constant.

6. The unmanned ship collision avoidance decision-making method according to claim 1, characterized in that: On the basis of the trained model, the model is tested through various encounter scenarios based on the test environment and the current basic information of the unmanned ship to verify the effectiveness of the model; wherein the surrounding environment information is uniformly represented by a vector as the model input for decision-making; In view of the problem that the surrounding environment of the unmanned ship will change constantly during sailing, the input information is processed into the same dimension by using GRU neural network to extract and retain the data information of each ship.

7. The unmanned ship collision avoidance decision-making method integrating the experience of ship officers according to claim 1, characterized in that: During the test, the sensors of the unmanned ship set a maximum detection range, and when the other ship exists within the detection range, it is judged whether to take avoidance measures; when the collision risk between the two ships exceeds the threshold, the collision avoidance decision is calculated and executed; and according to the distance between the current state of the unmanned ship and the target point, the reward is fed back to guide the decision-making behavior of the unmanned ship at the next moment; The model learns the action strategy by continuously interacting with the environment, and the learning effect is represented by the cumulative reward value of each training event, and the total reward value and the result are calculated; according to the test result and the feedback reward value, it is judged whether the unmanned ship reaches the target point.

8. An unmanned ship collision avoidance decision system fusing the experience of the ship's crew, based on a computer system, characterized by: The computer program instructions stored on the memory and capable of being executed by the processor, when the processor executes the computer program instructions, adopt the unmanned ship collision avoidance decision-making method integrating the experience of ship officers according to any one of claims 1-7 to construct the algorithm model obtained and trained, and output the unmanned ship collision avoidance decision.

Images