Ship interactive avoidance decision method in mixed navigation environment

Abstract

The application provides a ship interactive avoidance decision-making method in a mixed navigation environment, and relates to the technical field of intelligent ship autonomous navigation, and comprises the following steps: S1, defining the ship driver's driving style in the mixed navigation environment, and acquiring encounter situation parameters; S2, modeling the ship collision avoidance decision-making stage process into a stage game interactive model according to the encounter situation parameters; S3, formulating a ship avoidance sequence habit action priority table, and solving the Nash payoff equilibrium strategy of the stage game interactive model; S4, modeling the ship interactive collision avoidance process into a Markov game model according to the Nash payoff equilibrium strategy; and S5, inputting the encounter situation parameters into the Markov game model for Nash-Q learning, and obtaining the optimal avoidance action of each ship. The application considers the ships in the encounter scene as intelligent individuals with independent thinking and decision-making capabilities, and realizes the interactive collision avoidance and autonomous resolution of collision dangers of ships with different intelligent degrees in the mixed navigation scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent ship autonomous navigation technology, and more particularly to a ship interaction avoidance decision-making method in a mixed navigation environment. Background Technology

[0002] In recent years, leading countries in intelligent ship technology research and development have conducted real-world ship tests in specific encounter scenarios in actual sea areas. For a considerable period in the future, the seas will be characterized by a mix of autonomous surface vessels with varying degrees of autonomy and traditional manned vessels. Currently, most domestic and international research on ship collision avoidance neglects the influence of surrounding vessels, primarily treating them as obstacles of constant speed and direction. It fails to consider changes in the intentions of surrounding vessels and does not address the interaction between autonomous vessels and their surroundings, thus rendering it unsuitable for collision avoidance in mixed-traffic environments.

[0003] The collision avoidance process of ships is essentially a process of seeking to maximize one's own or collective interests in a conflict of interests where competition and cooperation coexist. Game theory is a theory that studies the interaction of strategic behaviors of rational decision-makers in a conflict of interests. Therefore, it is of great significance to use game theory to study the safe interaction and collision avoidance between intelligent ships and manned ships in mixed-traffic environments.

[0004] This invention addresses the collision avoidance problem in mixed-traffic environments by considering the navigation status of other vessels, analyzing the cooperation and competition mechanisms among vessels, and constructing a Markov game process that allows vessels to interact with each other in real time throughout the collision avoidance process, thus achieving safe collision avoidance for multiple vessels in mixed-traffic scenarios. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a ship interaction collision avoidance decision-making method in a mixed navigation environment. This invention focuses on the ship interaction collision avoidance problem under mixed manned and unmanned navigation conditions, and considers all ships as intelligent individuals with independent thinking and decision-making capabilities, in order to achieve the goal of autonomous ships interacting with manned ships to avoid collisions and autonomously resolve collision hazards.

[0006] The technical means employed in this invention are as follows:

[0007] A method for ship interaction avoidance decision-making in a mixed navigation environment includes the following steps:

[0008] S1. Define the ship handling style of the ship driver and the characteristics of heterogeneous ships in a mixed navigation environment, and obtain the encounter situation parameters in the mixed traffic flow.

[0009] S2. Based on the encounter situation parameters, the collision avoidance decision-making process of heterogeneous ships with different levels of autonomy is modeled as a stage game interaction model.

[0010] S3. Develop a priority table for ship avoidance sequence actions and solve the Nash payoff equilibrium strategy for the stage game interaction model.

[0011] S4. Based on the Nash payoff equilibrium strategy, the ship collision avoidance process is modeled as a Markov game model.

[0012] S5. The encountered situation parameters are input into the Markov game model for Nash-Q learning to obtain the optimal avoidance action for different levels of autonomy.

[0013] Furthermore, in S1, based on the different perceptions of the minimum safe encounter distance by ship operators, the ship operator's handling style is defined into three types: aggressive, normal, and conservative. The minimum safe encounter distance threshold for the aggressive type is 1 nautical mile, for the normal type it is 1.5 nautical miles, and for the conservative type it is 2 nautical miles. The characteristics of heterogeneous ships are mainly divided into manned ships and unmanned ships based on the different degrees of autonomy of the ships.

[0014] Furthermore, in S1, the situational parameters encountered in the mixed traffic flow include the driver's ship handling style type, ship heading, speed, bearing, and the DCPA, TCPA, relative distance, and relative bearing between ships.

[0015] Furthermore, in S2, the game interaction model is represented as {N,A,U,A}. * Let N be the game participants, A be the set of strategies, U be the game payoff, and A be the set of payoffs. * The Nash equilibrium is defined as follows: the game participants are ships with different levels of intelligence; the strategy set is the set of ship avoidance actions; the game payoff is the different payoff values ​​obtained by the participants based on different avoidance strategies according to the evaluation index; and the Nash equilibrium is the optimal combination of strategies adopted by all participants to maximize their driving benefits.

[0016] Furthermore, the game payoff evaluation indicators include legality payoff, safety payoff, economic payoff, and driving payoff;

[0017] The legal benefits are:

[0018]

[0019]

[0020] Among them, A i ∈A' i To comply with collision avoidance rules, Let i be the legitimate payoff of ship i and ship j at stage t in the game. Let be the average legitimate payoff of ship i at time t in the phase game;

[0021] The security benefits are as follows:

[0022]

[0023] in, Let q be the nearest safe meeting distance between vessel i and vessel j. i For the driving style of vessel i, d(q) i ) represents the psychological safety expectations of ship i;

[0024] The economic benefits are:

[0025]

[0026] Where Δψ is the change in decision course;

[0027] The driving benefits are:

[0028]

[0029] Where ρ is the weighting coefficient, ρ1+ρ2+ρ3=1.

[0030] Furthermore, in S3, the priority consideration factors for the ship avoidance sequence action are, in order, avoidance responsibility, driving style, and ship type. Among them, the priority action order for avoidance responsibility is, in order, giving way, equal, and straight course; the priority action order for driving style is, in order, conservative, normal, and aggressive; and the priority action order for ship type is, in order, manned ship and unmanned ship. The priority factors are constructed into a ship avoidance sequence action priority matrix. Based on the ship avoidance sequence action priority matrix, the order of ship interaction in the encounter scenario can be obtained.

[0031] Furthermore, in S3, the solution process for the Nash payoff equilibrium strategy in the stage game interaction is as follows:

[0032] p(A)=(E A ) -1 e A (a i,t )∈[0,1]

[0033] p(B) = (E) B ) -1 e B (b j,t )∈[0,1]

[0034] Where E represents the total future payoff of the ships, e represents the payoff of the ships taking collision avoidance actions, and a represents the basic set of ship actions. In the stage game, the two ships adopt the strategy corresponding to the maximum value of p(A) and p(B) (a... * ,b * This is the Nash equilibrium of the stage game.

[0035] Furthermore, S4 specifically includes: representing the ship-to-ship collision avoidance process based on Markov game theory as follows:

[0036] {n,S,A1,...,A n ,T,γ,R1,...,R n}

[0037] Where n represents the number of participating ships; S is the state set of the ships; A i It is the action set of the i-th ship; T is the state transition probability, representing the joint action (a1,...a2) taken in the current state. n The probability distribution of the next state at time ), where γ is the discount factor, γ∈[0,1]; R i This is the revenue of the i-th ship;

[0038] Nash Q is defined as the sum of the expected discounted returns of all ships maintaining their equilibrium strategy in the next phase. The Nash Q-function of ship i, and when all ships follow the Nash equilibrium strategy, in (s, a1, ... a n The Nash Q-function of ship i is defined as the sum of its current collision avoidance reward and its future rewards, i.e.:

[0039]

[0040] Where p(s′∣s,a1,…,a) n ) represents the probability that a ship will transition to state s′ after taking joint collision avoidance actions (a1,...an) in state s. For the joint Nash equilibrium strategy of ships, r i (s,a1,…,a n ) represents the ship i in state s and the joint collision avoidance action (a1,...a2). n The stage returns under ) Let represent the total discounted reward of ship i in an infinite number of phase games starting from state s′ when all ships follow the equilibrium collision avoidance strategy.

[0041] Since the Q-table only applies to the current moment, the Q-value is updated according to the following formula:

[0042]

[0043] in:

[0044] Furthermore, S5 specifically includes the following steps:

[0045] Run the Nash-Q learning algorithm, input the ship collision avoidance action sequence, the ship collision avoidance action set, and the ship payoff function. This serves as the ship's optimal equilibrium collision avoidance strategy after several stages of game theory. The ship's optimal equilibrium collision avoidance strategy is the ship's optimal avoidance action.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] This invention provides a ship interaction collision avoidance decision-making method in a mixed navigation environment. It mainly uses game theory to study the decision-making behavior of multiple ships with interactive collision avoidance capabilities in a mixed manned and unmanned navigation environment. It considers the navigation status of other ships in the surrounding area, analyzes the cooperation and competition mechanisms between ships, and enables each ship to meet the collision avoidance rules and safely encounter other ships. It realizes that each ship can adjust its own collision avoidance behavior strategy according to the behavior of other ships' drivers, interact with other ships in a human-like manner, and autonomously perform collision avoidance operations to resolve collision hazards. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the method of the present invention.

[0050] Figure 2 This is a geometric diagram illustrating the impact of different driver risk perceptions on collision avoidance decisions.

[0051] Figure 3 This is a schematic diagram of the ship interaction game collision avoidance of the present invention.

[0052] Figure 4 This is a diagram of the multi-agent reinforcement learning framework of the present invention.

[0053] Figure 5 This is a flowchart of the Nash-Q learning algorithm of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] This invention provides a method for ship interaction avoidance decision-making in a mixed navigation environment, comprising the following steps:

[0057] S1. Define the ship handling style of the driver in a mixed navigation environment and obtain the encounter situation parameters;

[0058] The collision avoidance decision-making mechanism for ships in mixed navigation scenarios is divided into three stages in this invention: a collision risk perception stage where ships share information and calculate collision risks during encounters; a collision avoidance decision-making stage where reasonable avoidance is performed based on collision avoidance rules; and a collision avoidance maneuvering stage based on the avoidance decision. To fully consider the individual differences among drivers of manned vessels in mixed navigation environments, their handling styles are categorized into three types based on their different perceptions of the minimum safe encounter distance: aggressive, normal, and conservative. The minimum safe encounter distance threshold is 1 nautical mile for the aggressive type, 1.5 nautical miles for the normal type, and 2 nautical miles for the conservative type. Encounter situation parameters include the driver's handling style type, ship's course, speed, bearing, and information such as DCPA, TCPA, relative distance, and relative bearing between ships.

[0059] S2. Based on the encounter situation parameters, the ship collision avoidance decision-making process is modeled as a stage game interaction model.

[0060] In this invention, the collision avoidance process between ships is viewed as a game-theoretic process seeking to maximize benefits amidst a conflict of interests involving both competition and cooperation. The game participants are ships with varying levels of intelligence; the strategy set is the set of ship avoidance actions, which, based on maritime practice, represents the selectable turning ranges of the ships; the game payoffs are the different payoff values ​​obtained by participants based on evaluation indicators such as legality, safety, and economy when adopting different avoidance strategies; and the Nash equilibrium state is the optimal combination of strategies adopted by all participants to maximize their driving benefits.

[0061] The game interaction model is represented as {N,A,U,A} * Let N be the game participants, A be the set of strategies, U be the game payoff, and A be the set of payoffs. * The Nash equilibrium is defined as follows: the game participants are ships with different levels of intelligence; the strategy set is the set of ship avoidance actions; the game payoff is the different payoff values ​​obtained by the participants based on different avoidance strategies according to the evaluation index; and the Nash equilibrium is the optimal combination of strategies adopted by all participants to maximize their driving benefits.

[0062] The description and modeling of ship interaction avoidance behavior based on game theory also includes: legitimacy benefits:

[0063]

[0064]

[0065] Where Ai∈A' i To comply with collision avoidance rules, Let i be the legitimate payoff of ship i and ship j at stage t in the game. Let be the average legitimate payoff of ship i at time t in the phase game;

[0066] The security benefits are:

[0067]

[0068] in, Let q be the nearest safe meeting distance between vessel i and vessel j. i For the driving style of vessel i, d(q) i ) represents the psychological safety expectations of ship i.

[0069] The economic benefits are:

[0070]

[0071] Where Δψ is the change in the decision course.

[0072] Driving benefits are:

[0073]

[0074] Where ρ is the weighting coefficient, ρ1+ρ2+ρ3=1.

[0075] S3. Develop a priority table of ship avoidance actions and solve the Nash payoff equilibrium strategy of the stage game interaction model.

[0076] The order of avoidance actions is determined based on different encounter situations and the participants' own attributes, as follows:

[0077]

[0078] The game process is represented as G = {N, M, U}, where N represents the players in the game; M represents the set of interactive actions when the players engage in collision avoidance; and U represents the payoff for each player to take a corresponding action.

[0079] The solution process for the Nash payoff equilibrium strategy in the phase game interaction is as follows:

[0080] p(A)=(E A ) -1 e A (a i,t )∈[0,1]

[0081] p(B) = (E) B ) -1 e B (b j,t )∈[0,1]

[0082] In the formula, E represents the total future payoff of the ships, e represents the payoff of the ships taking collision avoidance actions, and a represents the basic action set of the ships. In the stage game, the two ships adopt the strategy corresponding to the maximum value of p(A) and p(B) (a). * ,b * This is the Nash equilibrium of the stage game.

[0083] S4. Based on the Nash payoff equilibrium strategy, the ship collision avoidance process is modeled as a Markov game model.

[0084] Based on Markov game theory, the ship-to-ship collision avoidance process can be represented as:

[0085] {n,S,A1,...,A n ,T,γ,R1,...,R n}

[0086] Where n represents the number of participating ships; S is the state set of the ships; A i It is the action set of the i-th ship; T is the state transition probability, representing the joint action (a1,...a2) taken in the current state. n The probability distribution of the next state at time ), where γ is the discount factor, γ∈[0,1]; R i This is the revenue of the i-th ship;

[0087] The Nash Q value is defined as the sum of the expected discounted returns of all ships maintaining their equilibrium strategy in the next phase. The Nash Q-function of ship i, and when all ships follow the Nash equilibrium strategy, in (s, a1, ... a n The Nash Q-function of ship i is defined as the sum of its current collision avoidance reward and its future rewards, i.e.:

[0088]

[0089] Where p(s′∣s,a1,…,a) n ) represents the probability that a ship will transition to state s′ after taking joint collision avoidance actions (a1,...an) in state s. For the joint Nash equilibrium strategy of ships, r i (s,a1,…,a n ) represents the ship i in state s and the joint collision avoidance action (a1,...a2). n The stage returns under ) Let represent the total discounted reward of ship i in an infinite number of phase games starting from state s′ when all ships follow the equilibrium collision avoidance strategy.

[0090] Since the Q-table only applies to the current moment, the Q-value is updated according to the following formula:

[0091]

[0092] in:

[0093] S5. The encounter situation parameters are input into the Markov game model for Nash-Q learning to obtain the optimal avoidance actions for each ship.

[0094] The collision avoidance maneuvers that a ship can take are discretized into corresponding turning angles, denoted by A1,...,A1. n Let represent the set of possible collision avoidance actions that ships 1, 2, ..., n can take. At each stage t in the collision avoidance process, ship i determines its collision avoidance action based on the collision avoidance actions of itself and other ships in the previous stage t-1. At stage t, the encounter state is s, and the equilibrium collision avoidance strategy adopted by ships 1, 2, ..., n is denoted by (π). 1 (s)...π n (s)) is used to represent the collision avoidance actions of ships 1, 2, ..., n in stage t, respectively, using a1, ... a n Therefore, under the encounter state s, the expected return of ship i can be expressed as... The process then employs the Nash-Q learning algorithm, taking the ship collision avoidance action sequence, ship collision avoidance action set, and ship payoff function as input, and the optimal equilibrium collision avoidance strategy obtained after several stages of game theory as output, to obtain the final equilibrium collision avoidance strategy sequence. The optimal equilibrium collision avoidance strategy is the ship's optimal avoidance action.

[0095] like Figure 1The flowchart of a ship interaction avoidance decision-making method in a mixed navigation environment is presented. The invention is divided into five steps: First, quantifying the ship collision avoidance mechanism and the pilot's ship handling style under mixed navigation; second, describing and modeling the ship interaction avoidance behavior based on game theory; third, formulating collision avoidance sequence actions and solving the game Nash equilibrium; fourth, constructing a ship interaction avoidance expression paradigm based on multi-agent reinforcement learning; and fifth, solving the optimal avoidance actions for each ship based on Nash-Q learning.

[0096] like Figure 2 This invention provides a geometrical illustration of the impact of different risk perceptions among ship operators on collision avoidance decisions. Specifically, under the same encounter situation, different operators have different perceptions of the collision risk, leading to different collision avoidance actions. Three operators share the same perceived safe encounter distance—they all want to safely pass another vessel within 2n miles—but their different perceptions of collision risk result in inconsistent timing of their avoidance actions. The operator of ship 1, perceiving the collision risk at point A, needs to take a turn of magnitude 'a' to achieve a safe encounter within 2n miles; the operator of ship 2, perceiving the collision risk at point B, needs to take a turn of magnitude 'b' to achieve the same result; and the operator of ship 3, perceiving the collision risk at point C, needs to take a turn of magnitude 'c' to achieve the same result. This demonstrates that different perceptions of collision risk among ship operators lead to differences in the magnitude of their avoidance actions. To fully consider the individual differences among drivers of manned vessels in mixed navigation environments, the driving styles of these vessels are classified into three categories based on the drivers' different perceptions of collision avoidance risks: aggressive, normal, and conservative. The classification is based on different perceptions of the minimum safe encounter distance, with thresholds of 1 nautical mile, 1.5 nautical miles, and 2 nautical miles, respectively.

[0097] like Figure 3This diagram illustrates a collision avoidance game between ships. Human pilots assess the current collision risk and predict the reactions of other ships to their actions, thus adopting the optimal strategy within the constraints of collision avoidance rules. Other ships can observe the actions of the pilots of the ships that act first, such as changes in speed and course, and the degree of these changes, through radar, AIS, and other equipment, making decisions that are most advantageous to themselves. Regardless of whether a ship acts first or later, different driving styles will lead to different game outcomes. For example, the outcome of a game between a conservative ship A and an aggressive ship B might be that ship A, while adhering to the rules, uses a larger turning angle to achieve a safer encounter with ship B at a greater distance, while ship B sails straight to achieve greater economic benefits. For example, ship A is a conservative vessel. Within its action set, ship A can choose action 1, action 2, and action 3. As the yielding vessel, ship A will first take collision avoidance action. Considering that ship B will respond to its actions, ship A will assess the corresponding benefits and take the action most advantageous to itself. Ship B then takes action and can choose action 1, action 2, and action 3 from its action set to react to ship A's actions. Figure 3 This process is described using a dynamic game tree. The end of the game tree represents the payoff for each action sequence of the two ships, with the top and bottom representing the payoffs for ships A and B under this action sequence, respectively. All ship drivers are rational, therefore all ships act with the goal of maximizing their payoff. When ship A chooses action 1, ship B will choose action 2 in response, resulting in a payoff of (0.41, 0.75). When ship A chooses action 2, ship B will choose action 1 in response, resulting in a payoff of (0.35, 0.45). When ship A chooses action 3, ship B will choose action 2 in response, resulting in a payoff of (0.74, 0.75). Ship A, acting first, can rationally predict the reaction of ship B, acting later, to its actions. Therefore, ship A will choose action 3 as its optimal strategy at this point, as the payoff for action 3 is greater than that for actions 1 and 2. In conclusion, (Action 3, Action 2) is the optimal action combination for the two ships in this round of game interaction, and it is an equilibrium result.

[0098] like Figure 4This is a framework diagram for multi-agent reinforcement learning. Specifically, in multi-agent games, each agent's actions affect the rewards of other agents. This interdependence complicates the problem because each agent needs to consider the impact of other agents' actions on itself. The Nash-Q learning algorithm addresses this issue by maintaining the Q-value function of each agent. Each agent's Q-value function is a function of its own actions and the actions of other agents. At each time step, each agent uses its Q-value function to choose its next action and receives a reward based on its choice. The agent's Q-value function is updated based on its own actions and the actions of other agents to reflect the new rewards and environmental state. In real-world traffic environments, ships in the same scenario are all agents, each with its own decision-making process and interests. A ship's collision avoidance decision-making process is influenced not only by the environment but also by the behavior of other ships. This invention incorporates the interaction between other ships and the ship itself into a Markov decision process.

[0099] like Figure 5 The flowchart illustrates the Nash-Q learning algorithm. The algorithm takes a sequence of ship collision avoidance actions, a set of ship collision avoidance actions, and a ship payoff function as input, and outputs the optimal equilibrium collision avoidance strategy obtained after several stages of game theory. First, the initial state of the ship encounter is set, i.e., the ship encounter state at t=0 is s0, and the expected payoff of ship i is initialized. Next, at each stage t of the ship encounter, ships 1,2,…n travel from A1,…,A n The collision avoidance action is randomly selected, and each ship performs the selected collision avoidance action a1,…a2. n This allows us to obtain the next stage state s′. Ships 1, 2, ..., n can calculate their immediate rewards based on the ship interaction collision avoidance decision reward function. Simultaneously, based on the reward function of ship i in the next state s′... The Nash equilibrium of the collision avoidance actions taken by the ship during this phase can be calculated, and the resulting strategy is denoted as (π). 1 (s')...π n (s')), and calculate the corresponding profit. Given the immediate payoffs, equilibrium strategies, and corresponding rewards for each ship, their Q-values ​​and equilibrium collision avoidance strategies are updated using formula (4.11), where 0 ≤ γ < 1 is a discount factor. The closer the value is to 0, the greater the weight of immediate payoffs in the total reward; conversely, the closer the value is to 1, the more emphasis is placed on future returns. In the subsequent encounter phases t+1, t+2, ..., the above collision avoidance action changes and learning process are repeated until the payoff functions of each ship converge, resulting in the final equilibrium collision avoidance strategy sequence.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship interaction avoidance decision-making method in a mixed navigation environment, characterized in that, Includes the following steps: S1. Define the ship handling style of the ship driver and the characteristics of heterogeneous ships in a mixed navigation environment, and obtain the encounter situation parameters in the mixed traffic flow. S2. Based on the encounter situation parameters, the collision avoidance decision-making process of heterogeneous ships with different levels of autonomy is modeled as a stage game interaction model. S3. Develop a priority table for ship avoidance sequence actions and solve the Nash payoff equilibrium strategy for the stage game interaction model. S4. Based on the Nash payoff equilibrium strategy, the ship collision avoidance process is modeled as a Markov game model. S4 specifically includes: representing the ship-to-ship collision avoidance process based on Markov game theory as follows: in, Indicates the number of participating vessels; It is the ship's state set; It is the first The action set of a single vessel; It is the state transition probability, representing the current state. Take joint action The current state The probability distribution; As a discount factor, ; It is the first The revenue of each vessel; The Nash Q-value is defined as the sum of the expected discounted returns of all ships maintaining their equilibrium strategy in the next phase. The Nash Q-function of a ship is defined, and given that all ships follow a Nash equilibrium strategy, the Nash Q-function is defined in the state set. The definition above The Nash Q function of a ship is the sum of its current collision avoidance reward and its future rewards, i.e.: in, Indicates a ship In state and joint collision avoidance operations Next stage of returns; From state Take joint collision avoidance action After transitioning to state The probability of; For the joint Nash equilibrium strategy of ships; This indicates that when all ships follow a balanced collision avoidance strategy, the first... The ship from the state The sum of discounted rewards for the initial infinite number of phases of the game; Since the Q-table only applies to the current moment, the Q-value is updated according to the following formula: in: S5. The encountered situation parameters are input into the Markov game model for Nash-Q learning to obtain the optimal avoidance action for different levels of autonomy.

2. The ship interaction avoidance decision-making method for mixed navigation environments according to claim 1, characterized in that, In S1, based on the different perceptions of ship operators regarding the minimum safe encounter distance, the ship operator's handling style is defined into three types: aggressive, normal, and conservative. The minimum safe encounter distance threshold for the aggressive type is 1 nautical mile, for the normal type it is 1.5 nautical miles, and for the conservative type it is 2 nautical miles. The characteristics of heterogeneous ships are mainly divided into manned ships and unmanned ships based on the different degrees of autonomy of the ships.

3. The ship interaction avoidance decision-making method for mixed navigation environments according to claim 1, characterized in that, In S1, the encounter situation parameters in the mixed traffic flow include the driver's ship handling style type, ship heading, speed, bearing, and the DCPA, TCPA, relative distance, and relative bearing between ships.

4. The ship interaction avoidance decision-making method for mixed navigation environments according to claim 1, characterized in that, In S2, the game interaction model is represented as follows: , N For game participants, A For strategy set, U For the benefit of the game, The Nash equilibrium is defined as follows: the game participants are ships with different levels of intelligence; the strategy set is the set of ship avoidance actions; the game payoff is the different payoff values ​​obtained by the participants based on different avoidance strategies according to the evaluation index; and the Nash equilibrium is the optimal combination of strategies adopted by all participants to maximize their driving benefits.

5. The ship interaction avoidance decision-making method for mixed navigation environments according to claim 4, characterized in that, The game payoff evaluation indicators include legality payoff, safety payoff, economic payoff, and driving payoff; The legal benefits are: in, To comply with collision avoidance rules, For ships i With ships j In the phased game t Legitimate gains at that time For ships i In the phased game t The average legal payoff at any given time; The security benefits are as follows: in, For ships i With ships j The closest safe distance, For ships i driving style For ships i Psychological safety expectations; The economic benefits are: in, For the change in course for decision-making; The driving benefits are: in, These are the weighting coefficients. .

6. The ship interaction avoidance decision-making method for mixed navigation environments according to claim 1, characterized in that, In S3, the priority considerations for the ship avoidance sequence action are, in order, avoidance responsibility, driving style, and ship type. The priority action order for avoidance responsibility is, in order, giving way, equal, and straight course; the priority action order for driving style is, in order, conservative, normal, and aggressive; and the priority action order for ship type is, in order, manned ship and unmanned ship. These priority factors are used to construct a ship avoidance sequence action priority matrix. Based on the ship avoidance sequence action priority matrix, the game order of ships in the encounter scenario can be obtained.

7. The ship interaction avoidance decision-making method for mixed navigation environments according to claim 1, characterized in that, In S3, the solution process for the Nash payoff equilibrium strategy in the stage game interaction is as follows: Where E represents the total future payoff of the ships, e represents the payoff of the ships taking collision avoidance actions, and a represents the basic set of ship actions. In the phase game, the two ships take... , The strategy corresponding to the maximum value This is the Nash equilibrium of the stage game.

8. The ship interaction avoidance decision-making method for mixed navigation environments according to claim 1, characterized in that, S5 specifically includes the following steps: Run the Nash-Q learning algorithm, input the ship collision avoidance action sequence, the ship collision avoidance action set, and the ship payoff function. This serves as the ship's optimal equilibrium collision avoidance strategy after several stages of game theory. The ship's optimal equilibrium collision avoidance strategy is the ship's optimal avoidance action.

Images