Multi-ship cooperative autonomous collision avoidance method based on PMA-MADDPG

Abstract

The invention discloses a multi-ship cooperative autonomous collision avoidance method based on PMA-MADDPG. The method comprises the following steps that ship agents and the environment where the ship agents are located are modeled into a multi-agent Markov game system; designing a reward function of the multi-agent Markov game system into a multi-target comprehensive reward function; establishing a PMA-MADDPG model based on a policy-value network structure, and introducing a long and short-term memory network and a priority experience playback mechanism into the model; the PMA-MADDPG model is trained, and after a specified training round is reached, the PMA-MADDPG model after training is completed is obtained; and inputting the current local observation information into the trained PMA-MADDPG model by each ship intelligent body, generating an optimal collision avoidance action at the next moment, and executing a collision avoidance task according to the optimal collision avoidance action. According to the method, through multi-agent collaborative decision and time sequence state perception, the ship collision avoidance safety and the navigation rule conformity in a complex maritime scene are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous collision avoidance of ships, and in particular to a multi-ship collaborative autonomous collision avoidance method based on PMA-MADDPG. Background Art

[0002] With the rapid development of the shipping industry and the increasing density of maritime vessel traffic, autonomous collision avoidance technology has become crucial for ensuring navigation safety. Traditional collision avoidance methods rely primarily on the Convention on the International Regulations for Preventing Collisions at Sea, 1972 (COLREGs) and human experience. However, in complex multi-vessel interaction scenarios, these methods suffer from poor adaptability to dynamic environments and insufficient collaborative decision-making capabilities. In recent years, deep reinforcement learning (DRL) has demonstrated advantages in the field of autonomous decision-making, particularly the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which is capable of handling multi-agent collaborative decision-making problems.

[0003] However, existing methods still have shortcomings. Ships rely solely on local observations at the current moment and lack the ability to remember historical state sequences, resulting in inconsistent decision-making. Existing methods typically focus on single objectives, such as collision avoidance or path optimization, and struggle to simultaneously meet multiple requirements, such as COLREG compliance and heading stability. Traditional experience replay mechanisms employ uniform sampling, ignoring the importance of high-value samples and impacting model convergence speed and stability. Summary of the Invention

[0004] The present invention provides a multi-ship collaborative autonomous collision avoidance method based on a Prioritized and Memory-Aware Multi-agent Deep Deterministic Policy Gradient (PMA-MADDPG) algorithm based on prioritized replay and time series perception to overcome the above problems.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A multi-vessel collaborative autonomous collision avoidance method based on PMA-MADDPG includes the following steps:

[0007] S1. Model the ship agent and its environment as a multi-agent Markov game system and determine the system's sextuplet g= <I,S,{O i},{A i}, T, {R}>, the six-tuple includes: the set of ship agents I, the global state space S, the observation space O of the ship agents i , the action space A of the ship agent i , system state transfer function T and the instant reward function R of the ship agent;

[0008] S2. Designing the instantaneous reward function as a multi-objective comprehensive reward function, wherein the multi-objective comprehensive reward function includes: collision risk reward, guidance reward, COLREGs reward, heading angle correction reward, and arrival and collision reward;

[0009] S3. Based on the observation space and the action space, establish a PMA-MADDPG model based on a policy-value network structure; and train the PMA-MADDPG model.

[0010] S4. Introducing a long short-term memory (LSTM) network into the policy network of the PMA-MADDPG model to form a time-series-aware policy network for enhancing the ship agent's ability to memorize historical state observation information. The time-series-aware policy network selects actions for the ship agent based on the ship agent's current local state observation information and historical state observation information. By executing the actions and interacting with the environment, the next-moment state observation information and corresponding rewards are obtained.

[0011] S5. Receive and store the current action, next-moment state observation information, corresponding reward, and current-moment local state observation information of the ship agent through the experience buffer pool of the PMA-MADDPG model; introduce a prioritized experience replay (PER) mechanism in the experience buffer pool to update the sampling priority of experience samples based on the temporal difference error (TD Error) calculated by the value network;

[0012] S6. Samples in the experience buffer pool are sampled through the value network of the PMA-MADDPG model, and the temporal difference error is calculated by combining the actions of other ship agents and their state observation information; the temporal difference error is fed back to step S5, and a gradient signal is generated, and the gradient signal is used to update the temporal perception policy network; steps S4 to S6 are repeated until the set training round is reached and the training is completed;

[0013] S7. The ship agent inputs the current local observation information into the trained PMA-MADDPG model to obtain the optimal collision avoidance action of the ship agent at the next moment for performing the collision avoidance task.

[0014] Furthermore, the instant reward is designed as a multi-objective comprehensive reward function, and the total reward function R of the multi-objective comprehensive reward function is defined as:

[0015] R=R ri +R gu +R r +R θ +R co (1)

[0016] The meanings of the parameters in the formula are as follows:

[0017] S21. Define collision risk reward R ri , whose expression is:

[0018]

[0019] Where CPA is the closest encounter point distance; d c1 d c2 Representing two thresholds of collision risk, they are defined as follows:

[0020]

[0021] Where, L is the length of the ship; V max is the maximum speed of the ship; d c Provide a dynamic safety buffer zone for ships;

[0022] S22. Define the guidance reward R gu , whose expression is:

[0023]

[0024] Where, d i is the distance between the ship and the target point; d r is the distance threshold; Δθ i is the error angle between the current heading and the target direction;

[0025] S23. Define COLREGs reward R r , whose expression is:

[0026]

[0027] Where λ is the rule correlation coefficient; The angle of course adjustment for actions taken by the vessel; The maximum heading range allowed for adjustment;

[0028] S24. Define heading angle correction reward R θ , whose expression is:

[0029]

[0030] Where Δθ is the absolute value of the relative angle difference between the current heading of the ship and the heading of the target point;

[0031] S25. Define arrival and collision rewards R co , whose expression is:

[0032]

[0033] Where, d co is the distance between the two ships; d dis_c and d dis_cl for different distance thresholds.

[0034] Furthermore, the closest encounter point distance CPA is calculated, and its expression is:

[0035]

[0036] Where, D represents the distance between the two ships; Indicates the heading of the ship's intelligent body; Indicates the position of the own ship agent relative to the target ship agent.

[0037] Furthermore, the processing process of the long short-term memory network introduced by the policy network is as follows:

[0038] S31, receiving the time series feature X of the input observation layer of the strategy network through the long short-term memory network = {x t-L+1 ,x t-L+2 ,...,x t}; where L is the sequence length, x t Status information of the ship agent and other ship agents, including: ship agent heading, ship agent speed, relative distance to other ship agents, speed of other ship agents, relative position and rule list;

[0039] S32. Integrate and update the time series features with the integrated information of the long short-term memory network at the previous moment through the long short-term memory network. The updated expression is:

[0040]

[0041] Where h t is the integrated information; t is the given time step; h t-1 is the integrated information of the long short-term memory network at the previous moment; ct-1 is the memory unit; σ is the Sigmoid activation function; tanh is the hyperbolic tangent function; W f 、W i 、W c 、W o is the weight corresponding to each gate; b f 、b i 、b c 、b o is the bias term;

[0042] S33. Calculate the input of the next layer of network based on the integrated information through the long short-term memory network, and the expression is:

[0043] o t =f(h t ;ω) (10)

[0044] Where f is determined by the long short-term memory network; ω is the network parameter of the long short-term memory network.

[0045] Furthermore, the time series difference error is calculated by a loss function, and the expression of the loss function is:

[0046]

[0047] Where N is the batch size, which indicates the number of experience samples sampled in each round of training; i is the sample; s i is the state of the i-th sample; a i is the action of the i-th sample; Q a (s i ,a i ) is the prediction report of the current value network for the “state-action” pair; y i Represents the return value output by the target value network; where y i With Q a (s i ,a i ) i -Q a (s i ,a i )) is the timing difference error δ i .

[0048] Furthermore, the implementation steps for updating the sampling priority of the experience samples based on the temporal difference error calculated by the value network are as follows:

[0049] S51. Calculate the sample priority according to the time series difference error. The expression is:

[0050] P i =|δ i |+ε (12)

[0051] Where, P i represents the sample priority; ε is a constant to prevent the priority from being 0; δ i is the time series difference error of the sample; where |δ i The larger the value of |, the i The greater the probability of being selected;

[0052] S52. Calculate the sampling probability according to the sample priority, and its expression is:

[0053]

[0054] Where α is a hyperparameter used to control the degree of priority, and α = 0 is uniform sampling.

[0055] Furthermore, when generating the gradient signal, an importance sampling weight is introduced to correct the gradient update; the importance sampling weight is calculated based on the sampling probability, and its expression is:

[0056]

[0057] Where β is the parameter that controls the importance sampling correction; N is the experience replay buffer.

[0058] Beneficial effects:

[0059] The present invention proposes a multi-ship collaborative autonomous collision avoidance method based on PMA-MADDPG. Unlike the traditional MADDPG method that only relies on the current observation state for strategy decision-making, the present invention introduces a long-short-term memory network to enable the ship intelligent body to have the memory of historical states and time series modeling capabilities. During the strategy training process, the priority experience replay mechanism introduced by the present invention improves the training efficiency and sample utilization, and prioritizes the use of high-value experience subsets from the experience cache pool to strengthen strategy learning for key scenarios. In terms of reward functions, the present invention constructs a multi-goal-oriented comprehensive reward system covering multiple evaluation dimensions such as task completion, collision avoidance rules, heading adjustment, CPA safety assessment, and COLREGs compliance. Compared with the existing technology, the present invention shows improvements in strategy stability, time series modeling capabilities, sample utilization efficiency, and heading compliance. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0061] Figure 1 Flowchart of the multi-ship collaborative autonomous collision avoidance method based on PMA-MADDPG of the present invention;

[0062] Figure 2 This is a schematic diagram of a two-ship encounter model according to the present invention;

[0063] Figure 3 This is the PMA-MADDPG network structure diagram of the present invention;

[0064] Figure 4 This is a flow chart of the priority experience playback mechanism of the present invention;

[0065] Figure 5 This is the PMA-MADDPG collision avoidance flow chart of the present invention;

[0066] Figure 6 This is a design diagram of the reward function of the present invention;

[0067] Figure 7 This is a comparison chart of the reward values ​​of the present invention. DETAILED DESCRIPTION

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0069] This embodiment provides a multi-vessel collaborative autonomous collision avoidance method based on PMA-MADDPG. Figure 1 As shown, the following steps are included:

[0070] S1. Model the ship agent and its environment as a multi-agent Markov game system and determine the system's sextuplet g= <I,S,{O i},{A i}, T, {R}>, the six-tuple includes: the set of ship agents I, the global state space S, the observation space O of the ship agents i , the action space A of the ship agent i , system state transfer function T and the instant reward function R of the ship agent;

[0071] S2. Designing the instantaneous reward function as a multi-objective comprehensive reward function, wherein the multi-objective comprehensive reward function includes: collision risk reward, guidance reward, COLREGs reward, heading angle correction reward, and arrival and collision reward;

[0072] S3. Based on the observation space and the action space, establish a PMA-MADDPG model based on a policy-value network structure; and train the PMA-MADDPG model.

[0073] S4. Introducing a long short-term memory network into the policy network of the PMA-MADDPG model to form a time-series-aware policy network for enhancing the ship agent's ability to memorize historical state observation information; the time-series-aware policy network selects actions for the ship agent based on the ship agent's current local state observation information and historical state observation information; and interacting with the environment by executing the actions to obtain the next-moment state observation information and the corresponding reward.

[0074] S5. Receive and store the current action, next-moment state observation information, corresponding reward, and current-moment local state observation information of the ship agent through the experience buffer pool of the PMA-MADDPG model; introduce a priority experience replay mechanism in the experience buffer pool to update the sampling priority of the experience samples according to the temporal difference error calculated by the value network;

[0075] S6. Samples in the experience buffer pool are sampled through the value network of the PMA-MADDPG model, and the temporal difference error is calculated by combining the actions of other ship agents and their state observation information; the temporal difference error is fed back to step S5, and a gradient signal is generated, and the gradient signal is used to update the temporal perception policy network; steps S4 to S6 are repeated until the set training round is reached and the training is completed;

[0076] S7. The ship agent inputs the current local observation information into the trained PMA-MADDPG model to obtain the optimal collision avoidance action of the ship agent at the next moment for performing the collision avoidance task.

[0077] Specifically, the ship agent and the environment are first modeled as a multi-agent Markov game system, and the system sextuple (agent set, state space, observation space, action space, state transfer function, and immediate reward function) is defined. Its role is to establish the mathematical basic framework of distributed collaborative decision-making; the immediate reward function is designed into a multi-objective comprehensive reward function, and the safety distance is quantified by the collision risk reward. The guidance reward maintains the route efficiency, the COLREGs reward is embedded in the maritime rule logic, the heading angle correction reward optimizes the maneuvering smoothness, and the arrival / collision reward marks the task end state. Its role is to build a multi-objective optimization guide that drives the ship agent to avoid collisions in compliance.

[0078] Secondly, the PMA-MADDPG model is constructed, and a network architecture is used in which a policy network generates action instructions and a value network evaluates action benefits. The model training process is started, which serves as the core computing engine for building distributed decision-making and evaluation. A long-short-term memory network is embedded in the policy network to form a temporal perception policy network. The historical state observation information caching mechanism is used to enhance the continuous understanding of the dynamic environment, and the optimal action instructions are selected based on the current and historical states. This solves the problem of state misjudgment caused by instantaneous perception interruptions. A priority experience replay mechanism is introduced, and the temporal difference error calculated by the value network is used to dynamically evaluate the value of experience samples. A training optimization mechanism for prioritizing high-value samples is established, which accelerates the efficiency of policy learning in key collision avoidance scenarios. The temporal difference error is calculated through the value network, and the policy network gradient signal is generated by combining the actions and state information of other ships to drive the iterative update of network parameters, which realizes the collaborative optimization of multi-agent strategies.

[0079] Finally, based on the fully trained model, the real-time local observation information is input into the temporal perception strategy network, which outputs the optimal collision avoidance action instructions that comply with the COLREGs rules. Its function is to convert the learning results into real-time autonomous collision avoidance control.

[0080] Specifically, the collision avoidance process of the PMA-MADDPG model is as follows: Figure 5 As shown;

[0081] First, the left side of the figure shows two interactive information sources: the ship collision avoidance environment and the ship agent, representing the real-time dynamic interaction between the ship and the environment. The ship collision avoidance environment includes the position information, heading angle, velocity vector of each ship agent, and the state of other ship agents; the observation vector received by the ship agent serves as the input state of reinforcement learning;

[0082] Secondly, the middle part of the figure shows the LSTM network embedded in the policy network, which includes an input gate, a forget gate, and an output gate. The input gate combines the current state observation information with the integrated information of the LSTM network at the previous moment, and determines the update of the memory unit at the current moment through a nonlinear activation function. The forget gate controls the retention of historical information and determines whether the integrated information of the LSTM network at the previous moment is retained or forgotten. The output gate determines the content of the integrated information of the LSTM network at the current moment output to the Actor network. It is the result of the LSTM network fusing the current state observation information and historical information.

[0083] Again, the upper portion of the figure shows the policy network (Actor network) and the value network (Critic network). Both networks consist of an online network and a target network. Despite improvements to the long short-term memory network, the training process remains based on a centralized training and distributed execution framework, with each ship agent sharing global information during training.

[0084] In addition, the upper right corner of the figure shows the calculation formula of the value network loss function. The value function after value network training is fitted to calculate the time series difference error;

[0085] Finally, the lower part of the figure shows the prioritized experience replay mechanism and the normalized feedback path. The normalized feedback path normalizes the immediate rewards of the ship agent’s interaction with the environment to prevent scale differences from interfering with the stability of network training.

[0086] Specifically, such as Figure 6 As shown in Figure 2, the instantaneous reward of the multi-agent Markov game system is designed as a multi-objective comprehensive reward function, and the total reward function R of the multi-objective comprehensive reward function is defined as:

[0087] R=R ri +R gu +R r +R θ +R co (1)

[0088] The meanings of the parameters in the formula are as follows:

[0089] S21. Define collision risk reward R ri , whose expression is:

[0090]

[0091] Where CPA is the closest encounter point distance; d c1 d c2 Representing two thresholds of collision risk, they are defined as follows:

[0092]

[0093] Where, L is the length of the ship; V max is the maximum speed of the ship; d c Provide a dynamic safety buffer zone for ships;

[0094] In a specific embodiment, the reward is based on the CPA to evaluate whether there is a potential collision risk between any two ships at their current speed and heading. If the CPA is greater than a preset threshold, a positive reward is given, otherwise a penalty is given to encourage the agent to avoid potential conflict paths in advance;

[0095] S22. Define the guidance reward R gu , whose expression is:

[0096]

[0097] Where, d i is the distance between the ship and the target point; d r is the distance threshold; Δθ i is the error angle between the current heading and the target direction;

[0098] In a specific embodiment, the reward is used to determine whether each ship has successfully reached the target point. The judgment criteria include whether the Euclidean distance from the target point is less than a preset redundancy distance, and whether the difference between the current heading and the target heading is within an allowable range;

[0099] S23. Define COLREGs reward R r , whose expression is:

[0100]

[0101] Where λ is the rule correlation coefficient; The angle of course adjustment for actions taken by the vessel; The maximum heading range allowed for adjustment;

[0102] In a specific embodiment, this reward is used to evaluate the legality of collision avoidance behavior in multi-ship encounter scenarios based on COLREGs. First, a table of encounter rules (head-on encounter, overtaking, and crossing encounter) is constructed based on the current state. Then, based on the next ship's course change, it is determined whether the correct avoidance action has been taken. Correct execution of the rules will be rewarded, while failure to do so will be penalized, aiming to ensure that the learning strategy complies with international collision avoidance regulations.

[0103] S24. Define heading angle correction reward R θ , whose expression is:

[0104]

[0105] Where Δθ is the absolute value of the relative angle difference between the current heading of the ship and the heading of the target point;

[0106] In a specific embodiment, the reward is based on the deviation between the current heading of the ship and the azimuth of the target point to evaluate the rationality of its heading. If the ship's heading towards the target is relatively accurate, a positive reward is given; if the deviation is large, a penalty is given to encourage the ship to adjust its heading towards the target area;

[0107] S25. Define arrival and collision rewards R co , whose expression is:

[0108]

[0109] Where, d co is the distance between the two ships; d dis_c and d dis_cl for different distance thresholds;

[0110] In a specific embodiment, the reward implements collision and arrival detection between ships. By calculating the Euclidean distance and relative heading between any two ships, combined with the defined lateral danger distance and heading difference, it is determined whether a collision occurs and the corresponding penalty reward and collision flag are returned.

[0111] Specifically, such as Figure 2 As shown in the figure, the position coordinates, heading angle, velocity vector and other basic parameters of the own ship (OS) agent and the target ship (TS) agent are included. The expression for calculating the closest approach point distance CPA is:

[0112]

[0113] Where, D represents the distance between the two ships; Indicates the heading of the ship's intelligent body; Indicates the position of the own ship agent relative to the target ship agent.

[0114] Specifically, such as Figure 3 As shown in the figure, based on the observation space and action space of the multi-agent Markov game system, a PMA-MADDPG model based on the strategy-value network structure is established. This model expands on the traditional MADDPG and introduces a long short-term memory (LSTM) module to enhance the time series modeling capability, enabling dynamic memory of the ship's historical trajectory and strategy optimization. The strategy network consists of three parts: the input observation module, the long short-term memory network temporal encoding module, and the actor action decision module.

[0115] The processing process of the long short-term memory network introduced by the policy network is as follows:

[0116] S31, receiving the time series feature X of the input observation layer of the strategy network through the long short-term memory network = {x t-L+1 ,x t-L+2 ,...,x t}; where L is the sequence length, x t Status information of the ship agent and other ship agents, including: ship agent heading, ship agent speed, relative distance to other ship agents, speed of other ship agents, relative position and rule list;

[0117] S32. Integrate and update the time series features with the integrated information of the long short-term memory network at the previous moment through the long short-term memory network. The updated expression is:

[0118]

[0119] Where h t is the integrated information; t is the given time step; h t-1 is the integrated information of the long short-term memory network at the previous moment; c t-1 is the memory unit; σ is the Sigmoid activation function; tanh is the hyperbolic tangent function; W f 、W i 、W c 、W o is the weight corresponding to each gate; b f 、b i 、b c 、b o is the bias term;

[0120] S33. Calculate the input of the next layer of network based on the integrated information through the long short-term memory network, and the expression is:

[0121] o t =f(h t ;ω) (10)

[0122] Where f is determined by the long short-term memory network; ω is the network parameter of the long short-term memory network.

[0123] In a specific embodiment, the MADDPG policy network structure is improved by improving the feedforward multi-layer perceptron structure based on instantaneous state input. The static input layer in the original policy network is replaced with a time series encoder, so that the policy network has the ability to extract dynamic behavior from the state sequence. The expression after replacement is:

[0124]

[0125] Where, is the local observation of the i-th agent at time t; is a parameterized strategy; L is the length of the input sequence; is the hidden state of the LSTM network.

[0126] Specifically, the time series difference error is calculated by a loss function, and the expression of the loss function is:

[0127]

[0128] Where N is the batch size, which indicates the number of experience samples sampled in each round of training; i is the sample; s i is the state of the i-th sample; a i is the action of the i-th sample; Q a (s i ,a i ) is the prediction report of the current value network for the “state-action” pair; y i Represents the return value output by the target value network; where y i With Q a (s i ,a i ) i -Q a (s i ,a i )) is the timing difference error δ i .

[0129] Specifically, such as Figure 4 As shown in Figure 2, the core idea of ​​the priority experience replay mechanism is to guide the ship agent to focus on learning key samples by dynamically adjusting the priority of samples in the experience replay pool, thereby accelerating strategy convergence. The specific steps are as follows:

[0130] First, the information generated by each ship agent during its interaction with the environment is encapsulated into a typical four-tuple experience set in reinforcement learning and temporarily stored in the experience cache pool. The four-tuple experience set includes: current state observation information, selected action, immediate reward, and next state observation information;

[0131] Secondly, experience samples are sampled from the experience buffer pool. At the same time, the sampling probability of the experience samples is determined to be recalculated. When the number of training iterations reaches the preset batch threshold, the sampling probability is updated, otherwise sampling continues.

[0132] The implementation steps for updating the sampling priority of the experience samples based on the temporal difference error calculated by the value network are as follows:

[0133] S51. Calculate the sample priority according to the time series difference error. The expression is:

[0134] P i =|δ i |+ε (14)

[0135] Where, P i represents the sample priority; ε is a constant to prevent the priority from being 0; δ i is the time series difference error of the sample; where |δ i The larger the value of |, the i The greater the probability of being selected;

[0136] S52. Calculate the sampling probability according to the sample priority, and its expression is:

[0137]

[0138] Where α is a hyperparameter used to control the degree of priority. When α = 0, it is uniform sampling.

[0139] After the sampling probability is updated, it is further determined whether the Importance Sampling (IS) weight correction mechanism is enabled. This is determined based on whether the IS mechanism is enabled, whether the current importance sampling correction coefficient β is greater than 0, and whether there is a significant sampling bias. When the judgment conditions are met, the importance sampling weight is calculated based on the sampling probability. The importance sampling weight is used to correct the gradient update when generating the gradient signal.

[0140] Finally, after completing the update of the sampling priority and the sampling probability, the experience samples containing the sampling priority and the sampling probability are stored again in the experience buffer pool and participate in the subsequent network update as a training batch.

[0141] Specifically, when generating the gradient signal, an importance sampling weight is introduced to correct the gradient update; the importance sampling weight is calculated based on the sampling probability, and its expression is:

[0142]

[0143] Where β is the parameter that controls the importance sampling correction; N is the experience replay buffer.

[0144] In a specific embodiment, Figure 7 The figure shows the reward value changes in the three-ship collision avoidance scenario. The horizontal axis is the number of training episodes, and the vertical axis is the reward value for the corresponding round.

[0145] In the figure, the blue solid line shows the training reward changes of the baseline traditional method MADDPG model. The curve fluctuates violently, showing non-stationarity and reward oscillation, indicating that the strategy converges slowly and is prone to falling into local optimality or even degradation.

[0146] The red solid line represents the P-MADDPG model that introduces the priority experience replay mechanism. Its reward stability is somewhat improved compared to the baseline traditional method, but the overall reward limit is limited. This shows that although the priority experience replay mechanism improves sample utilization efficiency, it still has difficulty capturing the trajectory continuity characteristics of ships in dynamic collision avoidance without the ability to model time series.

[0147] The green solid line represents the PMA-MADDPG model. As can be seen from the curve trend, the model achieves rapid growth in the early stages of training and maintains a high reward region after approximately 1,000 episodes. Its fast convergence speed and low volatility verify the stability and generalization of its strategy.

[0148] In addition, the light red shaded area marks the reward range improved by the P-MADDPG model in the stable stage (about 1500 to 5000 episodes), and the light green shaded area represents the stable range of rewards improved by the PMA-MADDPG model during the entire training process.

[0149] The present invention has the following beneficial effects:

[0150] The present invention proposes a multi-ship collaborative autonomous collision avoidance method based on PMA-MADDPG. Unlike the traditional MADDPG method that only relies on the current observation state for strategy decision-making, the present invention introduces a long-short-term memory network to enable the ship intelligent body to have the memory of historical states and time series modeling capabilities. During the strategy training process, the priority experience replay mechanism introduced by the present invention improves the training efficiency and sample utilization, and prioritizes the use of high-value experience subsets from the experience cache pool to strengthen strategy learning for key scenarios. In terms of reward functions, the present invention constructs a multi-goal-oriented comprehensive reward system covering multiple evaluation dimensions such as task completion, collision avoidance rules, heading adjustment, CPA safety assessment, and COLREGs compliance. Compared with the existing technology, the present invention shows improvements in strategy stability, time series modeling capabilities, sample utilization efficiency, and heading compliance.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-vessel cooperative autonomous collision avoidance method based on PMA-MADDPG, characterized by: The following steps are involved: S1. Model the ship agent and its environment as a multi-agent Markov game system and determine the system's sextuplet g= <I,S,{O i },{A i }, T, {R}>, the six-tuple includes: the set of ship agents I, the global state space S, the observation space O of the ship agents i , the action space A of the ship agent i , system state transfer function T and the instant reward function R of the ship agent; S2. Designing the instantaneous reward function as a multi-objective comprehensive reward function, wherein the multi-objective comprehensive reward function includes: collision risk reward, guidance reward, COLREGs reward, heading angle correction reward, and arrival and collision reward; S3. Based on the observation space and the action space, establish a PMA-MADDPG model based on a policy-value network structure; and train the PMA-MADDPG model. S4. Introducing a long short-term memory network into the policy network of the PMA-MADDPG model to form a time-series-aware policy network for enhancing the ship agent's ability to memorize historical state observation information; the time-series-aware policy network selects actions for the ship agent based on the ship agent's current local state observation information and historical state observation information; and interacting with the environment by executing the actions to obtain the next-moment state observation information and the corresponding reward. S5. Receive and store the current action, next-moment state observation information, corresponding reward, and current-moment local state observation information of the ship agent through the experience buffer pool of the PMA-MADDPG model; introduce a priority experience replay mechanism in the experience buffer pool to update the sampling priority of the experience samples according to the temporal difference error calculated by the value network; S6. Samples in the experience buffer pool are sampled through the value network of the PMA-MADDPG model, and the temporal difference error is calculated by combining the actions of other ship agents and their state observation information; the temporal difference error is fed back to step S5, and a gradient signal is generated, and the gradient signal is used to update the temporal perception policy network; steps S4 to S6 are repeated until the set training round is reached and the training is completed; S7. The ship agent inputs the current local observation information into the trained PMA-MADDPG model to obtain the optimal collision avoidance action of the ship agent at the next moment for performing the collision avoidance task.

2. The multi-vessel collaborative autonomous collision avoidance method based on PMA-MADDPG according to claim 1 is characterized in that: The instant reward is designed as a multi-objective comprehensive reward function, and the total reward function R of the multi-objective comprehensive reward function is defined as: R=R ri +R gu +R r +R θ +R co (1) The meanings of the parameters in the formula are as follows: S21. Define collision risk reward R ri , whose expression is: Where CPA is the closest encounter point distance; d c1 d c2 Representing two thresholds of collision risk, they are defined as follows: Where, L is the length of the ship; V max is the maximum speed of the ship; d c Provide a dynamic safety buffer zone for ships; S22. Define the guidance reward R gu , whose expression is: Where, d i is the distance between the ship and the target point; d r is the distance threshold; Δθ i is the error angle between the current heading and the target direction; S23. Define COLREGs reward R r , whose expression is: Where λ is the rule correlation coefficient; The angle of course adjustment for actions taken by the vessel; The maximum heading range allowed for adjustment; S24. Define heading angle correction reward R θ , whose expression is: Where Δθ is the absolute value of the relative angle difference between the current heading of the ship and the heading of the target point; S25. Define arrival and collision rewards R co , whose expression is: Where, d co is the distance between the two ships; d dis_c and d dis_cl for different distance thresholds.

3. The multi-vessel collaborative autonomous collision avoidance method based on PMA-MADDPG according to claim 2 is characterized in that: The closest encounter point distance CPA is calculated as follows: Where, D represents the distance between the two ships; Indicates the heading of the ship's intelligent body; Indicates the position of the own ship agent relative to the target ship agent.

4. The multi-vessel collaborative autonomous collision avoidance method based on PMA-MADDPG according to claim 1 is characterized in that: The processing process of the long short-term memory network introduced by the policy network is as follows: S31, receiving the time series feature X of the input observation layer of the strategy network through the long short-term memory network = {x t-L+1 ,x t-L+2 ,...,x t }; where L is the sequence length, x t Status information of the ship agent and other ship agents, including: ship agent heading, ship agent speed, relative distance to other ship agents, speed of other ship agents, relative position and rule list; S32. Integrate and update the time series features with the integrated information of the long short-term memory network at the previous moment through the long short-term memory network. The updated expression is: Where h t is the integrated information; t is the given time step; h t-1 is the integrated information of the long short-term memory network at the previous moment; c t-1 is the memory unit; σ is the Sigmoid activation function; tanh is the hyperbolic tangent function; W f 、W i 、W c 、W o is the weight corresponding to each gate; b f 、b i 、b c 、b o is the bias term; S33. Calculate the input of the next layer of network based on the integrated information through the long short-term memory network, and the expression is: Where f is determined by the long short-term memory network; ω is the network parameter of the long short-term memory network.

5. The multi-vessel collaborative autonomous collision avoidance method based on PMA-MADDPG according to claim 1 is characterized in that: The time series difference error is calculated by a loss function, and the expression of the loss function is: Where N is the batch size, which indicates the number of experience samples sampled in each round of training; i is the sample; s i is the state of the i-th sample; a i is the action of the i-th sample; Q a (s i ,a i ) is the prediction report of the current value network for the "state-action" pair; y i Represents the return value output by the target value network; where y i With Q a (s i ,a i ) i -Q a (s i ,a i )) is the timing difference error δ i .

6. The multi-vessel collaborative autonomous collision avoidance method based on PMA-MADDPG according to claim 1, characterized in that: The implementation steps for updating the sampling priority of the experience samples based on the temporal difference error calculated by the value network are as follows: S51. Calculate the sample priority according to the time series difference error. The expression is: P i =|δ i |+e (12) Where, P i Indicates sample priority; ε is a constant to prevent priority from being 0; δ i is the temporal difference error of the sample; where |δ i The larger the value of |, the i The greater the probability of being selected; S52. Calculate the sampling probability according to the sample priority, and its expression is: Where α is a hyperparameter used to control the degree of priority, and α = 0 is uniform sampling.

7. The multi-vessel collaborative autonomous collision avoidance method based on PMA-MADDPG according to claim 1, characterized in that: Importance sampling weights are introduced when generating the gradient signal to correct the gradient update; the importance sampling weights are calculated based on the sampling probability, and the expression is: Where β is the parameter that controls the importance sampling correction; N is the experience replay buffer.

Images