A Multi-Agent Reinforcement Learning Collision Avoidance Method for Unmanned Surface Vessels Based on Policy Order Update

Abstract

This invention provides a multi-agent reinforcement learning-based unmanned surface vessel (USV) formation collision avoidance method based on policy order updates, belonging to the field of USV control technology. The model training phase includes: constructing a multi-USV formation collaborative decision-making environment and establishing a leader-follower topology; constructing local observation vectors for each vessel; constructing a multi-agent reinforcement learning decision-making model including a policy network and a value evaluation network; introducing a collision avoidance guidance mechanism based on vector field histograms, which dynamically switches the desired course based on the minimum obstacle distance and incorporates it into the observation vectors, while simultaneously switching the follower's target point from the formation's desired point to the leader's global target point during collision avoidance; and employing a policy order update mechanism: updating the parameters of each USV sequentially according to a preset order. The online execution phase includes: deploying the trained policy network, with each vessel independently outputting actions; when the original leader fails, selecting the follower closest to its global target point as the new leader.

Description

Technical Field

[0001] This invention belongs to the field of unmanned vessel control technology, specifically relating to a multi-agent reinforcement learning method for unmanned vessel formation collision avoidance based on policy order updates. Background Technology

[0002] With the continuous development of autonomous navigation technology for unmanned surface vessels (USVs), research focus has gradually expanded from single-vessel autonomous control to multi-USV swarm collaborative control. Compared to single-vessel systems, multi-USV swarms can perform complex tasks such as environmental monitoring, maritime patrol, target search, and cargo transportation over a larger sea area through distributed collaboration, offering advantages such as wider coverage, higher mission efficiency, and stronger system robustness. However, in real-world maritime environments, multi-USV swarms not only need to maintain a predetermined formation during navigation but also need to cope with collision risks caused by static obstacles, dynamic approaching vessels, and complex sea state disturbances. Therefore, how to achieve safe and efficient collaborative collision avoidance while maintaining the stability of the swarm structure has become a crucial research problem in the field of autonomous USV navigation.

[0003] Currently, in the field of multi-unmanned surface vessel (USV) formation control and collision avoidance, existing research mainly focuses on the following technical approaches: One approach is collision avoidance strategies based on geometric rules, artificial potential fields, and velocity barriers. These methods are characterized by clear structure and ease of implementation, but they typically rely on strong prior rule design and are prone to local optima, motion oscillations, or insufficient adaptability in complex dynamic scenarios. Another approach is formation control methods based on distributed consensus control, virtual structures, and leader-follower models. These methods have certain advantages in formation maintenance, but they often rely on relatively accurate dynamic models and have high requirements for communication quality and system modeling accuracy. When environmental uncertainty increases or obstacle interactions become more complex, their collision avoidance capability and robustness will significantly decrease. A third approach is collaborative decision-making methods based on optimization methods such as model predictive control. These methods can, to some extent, balance constraint handling and trajectory optimization, but their computational complexity is high, making it difficult to directly meet the needs of real-time collaborative decision-making among multiple USVs in complex dynamic sea conditions.

[0004] To reduce reliance on precise models and manual rules, multi-agent reinforcement learning methods have been increasingly introduced into multi-unmanned surface vessel (USV) formation control and cooperative collision avoidance tasks in recent years. These methods construct interaction mechanisms between agents and the environment, enabling each USV to learn formation maintenance and collision avoidance decision-making strategies during training and independently make action choices based on local observations during execution. Therefore, they show good application potential in adapting to complex environments. Existing research has applied multi-agent deep reinforcement learning to USV formation navigation, obstacle avoidance, and local cooperative control, improving the autonomous decision-making ability of multi-vehicle systems in complex environments through centralized training and distributed execution.

[0005] However, existing technologies still have the following core defects and shortcomings:

[0006] (i) Existing unmanned surface vessel (USV) formation decision-making methods based on traditional multi-agent reinforcement learning lack training stability and struggle to obtain reliable cooperative collision avoidance strategies. Existing methods typically employ a multi-agent synchronous update approach, which easily introduces environmental non-stationarity during training. Furthermore, they suffer from difficulties in credit allocation due to the inaccurate assessment of individual contributions under global rewards. This leads to situations where reward fluctuations are significant, policy oscillations are pronounced, or convergence to suboptimal solutions during policy updates, making it difficult to form stable and effective collaborative formation decision-making capabilities.

[0007] (ii) Existing methods struggle to coordinate the coupled objectives of formation keeping and collision avoidance, and are prone to generating ineffective actions in complex close-range interaction scenarios, affecting collision avoidance efficiency and formation stability. In complex scenarios with dense obstacles, limited local observation, or close encounters between multiple unmanned vessels, existing methods struggle to effectively balance target tracking, formation keeping, and collision avoidance behaviors, easily leading to frequent action switching, uncoordinated turning and speed adjustments, increased track redundancy, and decreased collision avoidance efficiency. Furthermore, they can cause formation disarray and insufficient coordination during collision avoidance, lacking a unified decision-making mechanism that balances safety, stability, and real-time performance.

[0008] (III) Existing methods lack adaptability and mission continuity under local observation conditions. In actual maritime environments, each unmanned vessel can usually only make decisions based on local environmental information and information of neighboring members. However, existing methods have limited ability to utilize local interaction relationships. Some methods rely on global state, fully connected communication, or fixed-size input structures. When the formation size changes, the strategy generalization ability is insufficient, making it difficult to adapt to the application requirements of dynamic changes in the number of multiple unmanned vessels. At the same time, when the leader fails, existing methods usually lack an effective formation reconstruction mechanism, making it difficult to maintain a unified target reference and formation structure in a timely manner, which seriously affects the system's robustness and mission continuity. Summary of the Invention

[0009] To address the shortcomings and deficiencies of existing technologies, this invention provides a multi-agent reinforcement learning-based unmanned surface vessel (USV) formation collision avoidance method based on policy sequence updates. This method aims to solve the core problems of existing USV formation collision avoidance schemes, such as insufficient training stability, difficulty in coordinating formation maintenance and collision avoidance objectives, weak adaptability to complex scenarios, and poor task continuity after leader failure. This invention employs a multi-agent reinforcement learning architecture with centralized training and distributed execution. First, a leader-follower formation topology is constructed, and a local observation vector containing the USV's own state, relative obstacle positions, and neighboring vessel interaction information is generated for each USV within the formation. A decision model consisting of a policy network and a value evaluation network is then built. The policy network processes variable-length obstacle and neighboring vessel information through an attention encoding module to improve formation-scale generalization ability. The value evaluation network uses a dual-commentator parallel structure to alleviate value overestimation. Simultaneously, a composite reward function adapted to formation maintenance and cooperative collision avoidance is designed to optimize cooperative decision-making performance. During training, a policy sequence update mechanism is used, updating the model parameters of each USV sequentially according to a preset order. When updating the policy of a single USV, the parameters of all other USVs are fixed, fundamentally reducing environmental non-uniformity caused by simultaneous multi-agent updates. This invention enhances stability, improves training convergence stability and strategy reliability, and introduces a collision avoidance guidance mechanism based on vector field histograms. It dynamically switches the desired course based on the minimum obstacle distance of the unmanned surface vessel (USV), integrating the switched desired course into the local observation vector. This forms a safe directional constraint on the action selection of the policy network. Combined with the dynamic switching of follower target points and velocity constraints during collision avoidance, it effectively coordinates the coupled goals of formation maintenance and cooperative collision avoidance, reducing ineffective actions in complex close-range interaction scenarios. During the online execution phase, each USV can independently complete distributed cooperative decision-making based solely on its own local observations, while simultaneously monitoring the leader's operational status in real time. When leader failure is detected, a new leader is selected based on the distance between candidate followers and the original leader's global target point. The new leader inherits the original task information, and the remaining followers reconstruct the formation target point, ensuring continuous execution of formation tasks and system robustness. This invention enables stable formation maintenance and safe, efficient cooperative collision avoidance for multi-USV formations in complex and dynamic marine environments, and can be widely applied to multi-USV cooperative operation scenarios such as marine environmental monitoring, patrol and search and rescue, and material transportation.

[0010] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0011] A multi-agent reinforcement learning-based unmanned surface vessel (USV) formation collision avoidance method based on policy order update includes a model training phase and an online execution phase, wherein:

[0012] The model training phase includes: constructing a multi-unmanned surface vessel (USV) formation collaborative decision-making environment; establishing a leader-follower formation topology; designating one USV as the leader to provide global navigation guidance, with the remaining USVs acting as followers; generating the formation's desired target point based on the leader's real-time pose; constructing a local observation vector for each USV within the formation, containing its own state information, relative obstacle position information, and neighboring vessel interaction information; and constructing a multi-agent reinforcement learning decision-making model based on the local observation vectors, including a policy network (corresponding to the actor network in subsequent embodiments) and a value evaluation network (corresponding to the critic network in subsequent embodiments). The policy network outputs continuous control actions based on the local observation vectors of a single USV, and the value evaluation network outputs continuous control actions based on the global state and... The entire formation's joint actions are evaluated for value; a reward function adapted to formation maintenance and cooperative collision avoidance is designed, and a collision avoidance guidance mechanism based on vector field histograms is introduced. By dynamically switching the expected course of the unmanned vessel according to the minimum obstacle distance, the switched expected course is integrated into the local observation vector. At the same time, when the unmanned vessel enters the collision avoidance state, the current task target point of the follower is switched from the formation's expected target point to the leader's global target point, thereby forming a safe direction constraint on the action selection of the policy network; the multi-agent reinforcement learning decision model is trained using a policy order update mechanism: in each training cycle, the model parameters of each unmanned vessel are updated sequentially according to a preset order. When updating the model parameters of a single unmanned vessel, the model parameters of all other unmanned vessels in the formation are fixed.

[0013] The online execution phase includes: deploying the trained policy network to each unmanned vessel; during online formation collision avoidance missions, each unmanned vessel independently outputs control actions based on real-time acquired local observation vectors; real-time detection of the formation leader's operational status; when the original leader fails, calculating the distance between each candidate follower and the original leader's current global target point, selecting the follower with the smallest distance as the new leader, the new leader inherits the original leader's global task information, and the remaining followers reconstruct the formation's desired target point based on the new leader's real-time pose, and continue to execute the formation collision avoidance mission.

[0014] Furthermore, in the strategy order update mechanism, the preset order is the update order generated by randomly sorting all unmanned vessels in the formation before the start of each training cycle. In each training cycle, the next training cycle is only entered after all unmanned vessels have completed one round of model parameter updates.

[0015] Furthermore, in the collision avoidance guidance mechanism based on vector field histograms, the specific rules for dynamically switching the desired course are as follows: when the minimum obstacle distance of the unmanned vessel is greater than the preset collision avoidance trigger threshold, the direction of the unmanned vessel's current mission target point is taken as the desired course; when the minimum obstacle distance is between the preset proximity safety threshold and the collision avoidance trigger threshold, the local collision avoidance reference course generated by the vector field histogram is taken as the desired course; when the minimum obstacle distance is less than or equal to the proximity safety threshold, the direction of the unmanned vessel's current mission target point is taken as the desired course; when the minimum obstacle distance of the unmanned vessel is less than the collision avoidance trigger threshold, a speed constraint is applied to the followers in the formation so that the real-time speed of the followers does not exceed the real-time speed of the current leader.

[0016] Furthermore, the detection of the formation leader's operational status specifically involves real-time detection of the leader's communication connection status with the other unmanned vessels in the formation. When the leader loses its communication connection with the formation, the original leader is deemed to have failed. The new leader inherits the original leader's global mission information, including the global navigation target point list and the current mission execution progress.

[0017] Furthermore, the policy network extracts features from the relative position information of obstacles and the interaction information of neighboring ships in the local observation vector through the attention encoding module. The extracted features are then concatenated with the network's own state information and input into the multilayer perceptron to output control actions. The value evaluation network adopts a dual-commentator parallel structure, with two identical sub-networks outputting independent action value evaluation results. When updating the model parameters, the minimum value of the two evaluation results is selected to participate in the target value calculation.

[0018] Furthermore, the reward function for the adaptive formation maintenance and cooperative collision avoidance is a weighted composite reward function, which includes formation maintenance reward, heading reward, obstacle safety distance reward, intra-formation safety distance reward and event-based reward and penalty, wherein the formation maintenance reward includes a continuous maintenance reward within the target tolerance range.

[0019] Furthermore, when performing formation collision avoidance missions online, each unmanned vessel completes distributed collaborative decision-making based solely on its own local observation vectors, without requiring global state information or full connectivity communication with the entire formation.

[0020] And, a multi-agent reinforcement learning unmanned surface vessel formation collision avoidance system, including a formation configuration module, a perception preprocessing module, a reinforcement learning decision module, a collision avoidance guidance module, a training control module and an online execution module;

[0021] The formation configuration module is used to construct a collaborative decision-making environment for multiple unmanned vessels, establish a leader-follower formation topology, and complete the role configuration of the leader and followers and the generation of formation objectives.

[0022] The perception preprocessing module is used to collect the self-state of each unmanned vessel, the relative position of obstacles and the interaction information with neighboring vessels, and to construct local observation vectors.

[0023] The reinforcement learning decision module includes a policy network unit and a value evaluation network unit, which are used to generate continuous control actions based on local observation vectors and complete the value evaluation during the intensive training phase.

[0024] The collision avoidance guidance module is used to dynamically switch the desired course based on the minimum obstacle distance of the unmanned vessel according to the vector field histogram algorithm, integrate the desired course into the local observation vector to complete the action constraint, and complete the dynamic switching of the follower target point in the collision avoidance state.

[0025] The training control module is used to execute the strategy order update mechanism and control the update order and update logic of the parameters of each unmanned surface vessel model.

[0026] The online execution module is used to deploy the trained policy network and complete the distributed formation collision avoidance task execution, leader state detection, and leader re-selection operation.

[0027] Furthermore, the training control module has a built-in random sorting unit, which is used to randomly sort all unmanned vessels in the formation before the start of each training cycle to generate the model parameter update order; the collision avoidance guidance module has a built-in threshold control unit and speed constraint unit, which are used to execute the segment switching logic of the desired course and the speed constraint of the follower.

[0028] Furthermore, the online execution module has a built-in distributed decision-making unit, which enables each unmanned vessel to generate control actions independently based on its own local observation vectors, without the need for global state information interaction.

[0029] And a computer system including a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement the method described above.

[0030] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0031] Compared to existing technologies, this invention and its preferred solution effectively address the problems of environmental non-stationarity and credit allocation difficulties introduced during synchronous updates in traditional multi-agent reinforcement learning through a multi-agent training mechanism that updates policies sequentially. This significantly reduces policy oscillations and reward fluctuations during training, improving the stability of policy convergence and the reliability of collaborative decision-making. Furthermore, through a collision avoidance constraint mechanism that deeply integrates vector field histogram guidance with reinforcement learning decision-making, stable coordination of the two coupled objectives of formation maintenance and collaborative collision avoidance is achieved. This effectively reduces ineffective actions and track redundancy in complex close-range interaction scenarios, improving collision avoidance efficiency and formation stability in complex obstacle environments. Finally, through distributed decision-making oriented towards local observations… The architecture and leader dynamic re-election mechanism significantly reduce the reliance of the formation system on global state information and fully connected communication. It can not only adapt to application scenarios with dynamically changing formation size and improve the environmental adaptability of the strategy, but also quickly complete formation reconstruction in the event of leader failure, maintain a unified target reference and formation structure, ensure the continuous execution capability of navigation missions, and significantly improve the overall robustness of the system. At the same time, the preferred solution of this invention, through the design of a composite reward function adapted to formation collaborative tasks, a dual-commentator parallel value evaluation structure, attention encoding feature extraction and other supporting optimizations, can further enhance the collaborative decision-making capability of multiple unmanned vessels and better adapt to the operational requirements of complex and dynamic real-world maritime environments. Attached Figure Description

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0033] Figure 1 This is a flowchart of the unmanned vessel formation collision avoidance method according to an embodiment of the present invention;

[0034] Figure 2 This is a structural diagram of an unmanned vessel formation according to an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of a collision avoidance guidance mechanism based on a vector field histogram according to an embodiment of the present invention;

[0036] Figure 4 This is a structural diagram of the multi-agent reinforcement learning framework according to an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of the actor network structure according to an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of the critic network structure according to an embodiment of the present invention;

[0039] Figure 7 A diagram illustrating invalid actions during formation collision avoidance. Detailed Implementation

[0040] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0041] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0042] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0043] To address the shortcomings of existing technologies in multi-unmanned surface vessel (USV) swarm collaborative collision avoidance, including insufficient training stability, difficulties in coordinating formation maintenance and collision avoidance, poor adaptability to complex scenarios, limited scalability, numerous invalid actions in complex close-range interaction scenarios, and insufficient mission continuity in the event of leader failure, this invention proposes a novel USV swarm collision avoidance method to achieve efficient formation maintenance and safe collaborative collision avoidance among multiple USVs in complex marine environments. By constructing a decision-making mechanism oriented towards swarm collaboration tasks, the method enhances the autonomous decision-making ability, collaborative collision avoidance capability, and mission execution stability of the multi-USV system in dynamic and complex scenarios.

[0044] This invention proposes a multi-agent reinforcement learning-based unmanned surface vessel (USV) formation collision avoidance method. By constructing a policy order update mechanism, it improves the stability of the multi-agent collaborative training process; by introducing a vector field histogram guidance and collision avoidance constraint collaboration mechanism, it reduces ineffective actions in complex scenarios and improves collision avoidance efficiency; and by designing a leader re-selection mechanism, it enhances the structural robustness and continuous task execution capability of the formation system under local observation and abnormal conditions, thereby achieving stable, efficient, and reliable collision avoidance for USV formations in complex dynamic environments.

[0045] This invention constructs a holistic methodological framework for collaborative decision-making among multiple unmanned vessels, the key points of which include the following three aspects:

[0046] (1) The multi-agent training mechanism based on policy sequential update is to update the policies of each unmanned vessel sequentially during the training process, so that when updating the policy of a single agent, the policies of the other agents remain unchanged, thereby reducing the environmental non-stationarity caused by the synchronous update of multiple agents; by reducing the mutual interference between policies, the policy update process is made more stable, and the consistency and reliability of training convergence are improved.

[0047] (2) The collision avoidance decision mechanism guided by the vector field histogram (VFH) is integrated. When the unmanned vessel approaches an obstacle, a reference heading is generated based on environmental perception information. The reference heading is introduced into the decision-making process through a distance threshold triggering mechanism to constrain the action selection of the reinforcement learning strategy, so that it prioritizes searching within a safe direction range, thereby reducing ineffective turning and repeated adjustment behavior, and improving the collision avoidance efficiency in complex scenarios.

[0048] (3) A formation robust mechanism for local observation conditions and leader failure scenarios: By dynamically sorting each unmanned vessel and re-selecting the leader based on the target point relationship during operation, the remaining unmanned vessels can still maintain a unified target reference when the leader fails, thereby avoiding formation structure disorder and ensuring the continuous execution capability of formation tasks.

[0049] Under the synergistic effect of the above mechanisms, each unmanned vessel makes distributed decisions based on local observation information. Through cyclical execution of state acquisition, information calculation, strategy decision-making and control execution, it achieves formation maintenance and cooperative collision avoidance.

[0050] The implementation of the present invention will be shown and described in detail below with reference to the accompanying drawings and through more specific embodiments.

[0051] The design and implementation process of this embodiment is as follows: Figure 1 As shown, the specific steps are as follows:

[0052] Step 1: Construct a multi-unmanned vessel formation mission scenario;

[0053] Step 2: Establish status and local observation information for multiple unmanned vessels;

[0054] Step 3: Construct a multi-agent reinforcement learning decision-making model;

[0055] Step 4: Design the reward function and constraint mechanism;

[0056] Step 5: Introduce a collision avoidance guidance mechanism;

[0057] Step 6, Multi-agent training and policy update;

[0058] Step 7: Output the trained formation collision avoidance strategy;

[0059] Step 8: Perform online formation collision avoidance missions for multiple unmanned vessels;

[0060] Step 9: Determine if the leader has failed;

[0061] Step 10: When the leader fails, the leader re-selection mechanism is initiated and the formation task continues.

[0062] For step 1, constructing a multi-unmanned vessel formation mission scenario, the specific steps are as follows:

[0063] Step 1.1: Construct a two-dimensional waterborne mission space, including multiple unmanned vessels, target points, obstacles, and environmental boundaries. Assume the formation size is... , for the first in the formation A single unmanned vessel establishes a ground-fixed coordinate system. with volume coordinate system Its in The pose vector is denoted as ,in Indicates location, Indicates the heading angle. Its position is... The velocity vector below is denoted as ,in , , These represent the longitudinal velocity, lateral velocity, and yaw rate, respectively. The transformation between the two coordinate systems is achieved using a rotation matrix. Description, which is defined as:

[0064]

[0065] Step 1.2, establish the dynamic equations of the unmanned vessel. Assuming the unmanned vessel is a rigid body and neglecting nonlinear hydrodynamic terms, its dynamic and kinematic equations are expressed as follows:

[0066]

[0067]

[0068] in,

[0069]

[0070]

[0071]

[0072] in, For the mass and additional mass matrix, The Coriolis and centrifugal force matrix, It is a linear damping matrix; For including longitudinal thrust With yaw moment The control input vector; This indicates external disturbances such as wind, waves, and ocean currents. The subscripts u, v, and r correspond to disturbances in the longitudinal, lateral, and bow roll directions, respectively. b u,i (t) represents the amount of external disturbance in the longitudinal direction for the i-th unmanned vessel. bv,i (t) represents the amount of external disturbance to the i-th unmanned vessel in the lateral direction. b r,i (t) represents the external disturbance of the i-th unmanned vessel in the bow roll direction, m1 and m2 represent the mass and added mass parameters respectively, and J is the moment of inertia.

[0073] Step 1.3: Establish a multi-agent collaborative decision-making environment. Define the environment as a Markov game:

[0074]

[0075] in, Represents a set of intelligent agents. Represents the global state space. and They represent the first The local observation space and action space of an intelligent agent. Represents the state transition probability. This represents the formation reward function. This represents the discount factor. USV at time Local observations are denoted as The global state is denoted as Joint actions are recorded as ,in Indicates the first An intelligent agent at time The actions. Accordingly, joint strategies. It can be written as:

[0076]

[0077] in, Represents the set of local observations of all agents. Indicates the first The policy parameters of each agent. The optimization objective of the formation task. Defined as:

[0078]

[0079] in, [] indicates a strategy The cumulative reward for all possible generated navigation paths is calculated using the expected value.

[0080] Step 1.4: Establish a leader-follower formation. Designate one unmanned surface vessel (USV) in the formation as the leader, and the remaining USVs as followers. The leader is responsible for guiding the overall formation path, while the followers are responsible for maintaining formation and coordinating collision avoidance. The leader's pose is represented as follows: , No. The position of the follower is For the first The target point position of each follower is generated in real time based on the leader's current position and heading, and is denoted as: the target point position of the follower formation. .

[0081] Step 1.5, calculate the formation error. The positional error of a follower is defined as:

[0082]

[0083] in, The current position of the follower.

[0084] When the follower has not yet reached its target point, its heading error is defined as:

[0085]

[0086] Once the follower reaches the target point, its heading error switches to:

[0087]

[0088] like Figure 2 As shown, this invention employs a leader-follower formation topology, using a single leader UAV as the formation's pose reference and global navigation guidance. The remaining follower UAVs use the leader's current position and heading as a reference, generating their respective desired target points within the formation in real time. In the figure, the leader's pose provides a unified heading and position reference for the formation. The formation position error of the i-th and k-th followers is the Euclidean distance between their current positions and their corresponding desired target points. The heading error is defined, depending on whether a follower has reached its desired formation target point, as the difference between its current heading and the target point direction, or between its current heading and the leader's heading. This standardized formation structure achieves unified formation and collaborative control among multiple UAVs.

[0089] For step 2, establishing the status and local observation information of multiple unmanned vessels is specifically divided into the following steps:

[0090] Step 2.1, define the action space. (The rest of the text appears to be a fragment and requires further context for accurate translation.) Unmanned boats at all times The action is defined as a two-dimensional continuous vector:

[0091]

[0092] in, and These represent the normalized control commands for longitudinal thrust and yaw moment, respectively. During the execution phase, the normalized commands are converted into physical control quantities through a linear mapping. and :

[0093]

[0094]

[0095] in and These represent the maximum available control inputs for the unmanned vessel in the longitudinal and yaw directions, respectively.

[0096] Step 2.2, establish local observation vectors. During the distributed execution phase, the... The local observation vector of the unmanned surface vessel is denoted as:

[0097]

[0098] in, For their own information, For obstacle information, Information for neighboring ships.

[0099] Step 2.3, Establish your own information. Personal Information The attributes, heading error, and current speed of an unmanned surface vessel are characterized by:

[0100]

[0101] in, Indicates the role of the unmanned vessel. Indicates the leader, It indicates a follower. and Indicates the target point relative to the first The unmanned vessel's position deviation in the x and y directions is measured by the environmental side length. Normalize. At the current speed, This is the maximum speed. The deviation between the current heading and the desired heading is defined as:

[0102]

[0103] In the formula, Indicates the first Unmanned boats at all times The expected course.

[0104] Step 2.4, establish the current task target point. (Note: The original text contains some inconsistencies and unclear punctuation. A more accurate translation would require the full context.) Unmanned boats at all times The current mission objective is Then we have:

[0105]

[0106] Among them, for leaders are:

[0107]

[0108] For followers:

[0109]

[0110] in, This indicates that the target point generated by the followers during the formation maintenance phase is based on the leader's position and heading. Indicates the current leader's goal. For the first Unmanned boats at all times The minimum obstacle distance, The threshold for collision avoidance.

[0111] Step 2.5, establish obstacle information. Obstacle Information From the most recent The relative positions of the obstacles constitute:

[0112]

[0113] in, and Indicates the first The selected obstacle is relative to the first The relative positions of the unmanned vessels. This refers to the detection radius of the unmanned vessel.

[0114] Step 2.6, Establish neighboring vessel information. Neighboring vessel information Used to express local interaction relationships within a formation, it is defined as follows:

[0115]

[0116] in, For the first Unmanned boats at all times The neighboring ships gathered. and Indicates the neighboring ship relative to the first The unmanned vessel's positional deviation; Threshold for information exchange with neighboring vessels; This represents the difference between the bow direction of the adjacent vessel and its own bow direction.

[0117] Step 2.7, Establish the global state. During the intensive training phase, the commentator network has access to the global state, which consists of local observations from all unmanned vessels:

[0118]

[0119] For step 3, constructing a multi-agent reinforcement learning decision model, the specific steps are as follows:

[0120] Step 3.1, construct the Actor network. Leaders and followers share the same network structure. The Actor network is based on the... Unmanned boats at all times Local observation Generate continuous control actions. Input observations are represented as follows:

[0121]

[0122] Among them, obstacle information Information with neighboring ships Feature extraction and self-state information are performed separately through the attention encoding module. The features are concatenated with the output features of the attention module and then input into a multilayer perceptron. The output features are the mean and standard deviation parameters of the Gaussian policy distribution, thereby defining the continuous policy. As a further preferred implementation, the attention encoding module adopts a single-head self-attention Transformer encoder structure, which independently encodes obstacle information and neighbor ship information. During the encoding process, the features of the corresponding sequence are used as the query, key, and value inputs. After linear embedding, Transformer encoding layer, and mean pooling, a fixed-dimensional feature vector is output to adapt to different numbers of obstacle and neighbor ship inputs, thereby improving the strategy generalization ability in scenarios with dynamically changing formation size.

[0123] like Figure 5 As shown, the actor network of this invention adopts a modular encoding-fusion structure, with the leader and followers sharing a completely consistent network structure. The network input is processed in three parallel paths: the first path is the unmanned vessel's own state information, directly input to the backend feature fusion layer; the second path is variable-length obstacle information, which undergoes linear embedding and feature extraction via an obstacle attention module, generating fixed-dimensional obstacle features through mean pooling; the third path is variable-length neighbor vessel information, which undergoes linear embedding and feature extraction via a neighbor vessel attention module, generating fixed-dimensional neighbor vessel features through mean pooling. After the three features are concatenated, they are input into two 256-dimensional ReLU-activated multilayer perceptrons, ultimately outputting the mean and standard deviation parameters of a continuous Gaussian policy distribution, defining the corresponding normalized continuous control policy for the unmanned vessel.

[0124] Step 3.2: Construct the Critic Network. During the intensive training phase, the Critic Network uses global information to evaluate the value of collaborative decisions. Its input consists of the corresponding global state. and joint actions Composition. A dual-Critic structure is used to approximate the two action value functions respectively. and During the strategy update process, the minimum of the two values ​​is used to calculate the target value.

[0125] like Figure 6 As shown, the critic network of this invention employs a dual-Q network structure, used only during the intensive training phase, for value evaluation of the joint actions of the entire formation. The network input consists of the global state composed of local observations from all unmanned vessels and the joint actions of the entire formation. After feature fusion, these are input in parallel into two Q-networks with identical structures. Both networks use two layers of 256-dimensional ReLU-activated multilayer perceptrons, outputting two independent action value functions, Q1 and Q2, respectively. During policy update, the minimum of the two Q values ​​is used to calculate the target Q value. Combined with an entropy regularization term, the target value is constructed. The dual-Q network structure effectively alleviates the Q-value overestimation problem during reinforcement learning training, improving the accuracy of value evaluation and the stability of policy training.

[0126] For step 4, designing the reward function and constraint mechanism involves the following steps:

[0127] Step 4.1, establish the weighted reward function. For the first... An unmanned boat, at any time The instant reward is recorded as:

[0128]

[0129] in, , , , These correspond to constraints on formation maintenance, heading, obstacle safety distance, and intra-formation safety distance, respectively. This represents the sum of event-based rewards for collisions, boundary crossings, reaching the target, and timeouts. This is the scaling factor.

[0130] Step 4.2, establish the formation maintenance reward. The formation maintenance expression is:

[0131]

[0132] in, For position reward weighting, This is the error penalty coefficient. Let be the formation position error at time t-1.

[0133] For the follower, when its target error at time t... Entering the target tolerance range Subsequently, a continuous reward system is adopted to maintain the formation, which is defined as:

[0134]

[0135] in, This serves as the base value for continuous rewards.

[0136] Add the reward to Therefore, the reward for maintaining the formation of followers can be written as:

[0137]

[0138] Step 4.3, establish heading reward. Heading reward is defined as:

[0139]

[0140] in, For heading reward weighting.

[0141] Step 4.4, establish obstacle safety distance rewards. Obstacle safety distance is defined as:

[0142]

[0143] in, Weighting of obstacle safety distance reward The minimum distance to the obstacle, This is the near-range safety threshold.

[0144] Step 4.5, establish an intra-formation safety distance reward. The intra-formation safety distance reward is defined as:

[0145]

[0146] in, Weighting of safety distance bonus within formation. Indicates time Next Ship and the The Euclidean distance between USVs Formation safety distance threshold.

[0147] Step 4.6, establish event-based rewards. Event-based rewards are uniformly represented as follows:

[0148]

[0149] in, These correspond to collision penalties, boundary crossing penalties, target achievement rewards, and timeout penalties, respectively. The values ​​of these event-based rewards and penalties can be adaptively adjusted by those skilled in the art based on the actual mission scenario, unmanned surface vessel performance parameters, and mission difficulty requirements.

[0150] Step 4.7, establish a speed constraint mechanism. When the... An unmanned vessel enters a collision avoidance state, which satisfies the condition. If the unmanned vessel is a follower, then the following speed constraint is applied:

[0151]

[0152] in, Indicates a group of followers. Let $\mathbf$ be the leader's current velocity. Equivalently, the follower's constrained velocity can be written as:

[0153]

[0154] For step 5, a collision avoidance guidance mechanism is introduced, which is specifically divided into the following steps:

[0155] Step 5.1, calculate the target direction. For the first... An unmanned boat, at any time From its current location relative to the current target point The direction of the target point can be obtained:

[0156]

[0157] Step 5.2: Construct the ray set and calculate the minimum obstacle distance. Based on the detection radius. The environmental perception results within are constructed at angular resolution. A uniformly divided set of rays:

[0158]

[0159] Where K is the total number of omnidirectional rays, which is determined by the detection range and angular resolution.

[0160] And calculate the nearest obstacle distance in the direction of each ray. . No. USV at time The minimum obstacle distance is defined as:

[0161]

[0162] in, The distance to the nearest obstacle along the direction of the k-th ray, min k This indicates that the minimum obstacle distance is calculated for all rays k.

[0163] Step 5.3, calculate the VFH cost function. When the minimum obstacle distance... Less than VFH trigger threshold At this time, the VFH module is activated to generate a local collision avoidance reference heading. The current target direction is... Then each direction The comprehensive cost function is defined as:

[0164]

[0165] in, , , These represent the target consistency weight, obstacle distance weight, and directional inertia weight, respectively. Describe the magnitude of change in the current direction relative to the optimal direction at the previous moment. The `wrap()` function is the indicator function, and it's the angle wrapping normalization function used to map the input angle difference to... The main value interval is used to obtain the minimum absolute angle between the two headings.

[0166] Step 5.4, determine the local collision avoidance reference direction. This is done by calculating the optimal collision avoidance direction index:

[0167]

[0168] This yields the corresponding local collision avoidance reference direction:

[0169]

[0170] Step 5.5, determine the desired course. A USV unmanned vessel at a time The expected course is defined as:

[0171]

[0172] Based on this, the heading deviation term in local observations is uniformly represented as:

[0173]

[0174] like Figure 3 As shown, the VFH collision avoidance guidance mechanism of this invention centers on the unmanned surface vessel (USV) and constructs an omnidirectionally uniform ray angle set based on the maximum detection radius R. This divides the space surrounding the USV into three control zones, enabling a smooth switch between formation maintenance and collision avoidance behavior: when the minimum obstacle distance is greater than the VFH trigger threshold... At this time, the unmanned vessel is in formation keeping mode, directly taking the target point direction as the desired course; when the minimum obstacle distance is within the close-range safety threshold... and When the minimum obstacle distance is less than or equal to the specified distance, the VFH module is activated, generating the optimal local collision avoidance reference heading through obstacle distance calculation using omnidirectional rays and solution of the comprehensive cost function; when the minimum obstacle distance is less than or equal to the specified distance... At this time, the unmanned vessel enters emergency collision avoidance mode, and completes the collision avoidance maneuver by re-establishing the target point direction as the reference and combining it with strong collision penalty constraints.

[0175] For step 6, multi-agent training and policy update, the specific steps are as follows:

[0176] Step 6.1, establish experience buffers. The system maintains two types of experience buffers:

[0177] 1) The global buffer is used to store... .

[0178] 2) Local buffers are used to store... .

[0179] Step 6.2 employs a sequential policy update mechanism. In each outer training loop, the agent updates sequentially. Update the strategy sequentially. When updating the first... Parameters of an unmanned ship At that time, the strategies of the other agents Updates are paused.

[0180] Step 6.3, update the soft critic network. For the first... An intelligent agent, whose soft critic network objective function is defined as:

[0181]

[0182] in, For the critic's network parameters, As an experience buffer, For the target Q value, The action value function output by the commentator network.

[0183] Where the target value for:

[0184]

[0185] This is the entropy coefficient.

[0186] Definition of joint action in the next moment:

[0187]

[0188] In the formula, except for the i-th unmanned vessel, the strategy parameters of all other unmanned vessels remain fixed to adapt to the strategy order update mechanism.

[0189] Step 6.4, update the actor network. The agent's policy is updated by minimizing the following objective:

[0190]

[0191] in, Let i be the parameters of the actor network for the i-th unmanned ship. This indicates that during the sequential update process, except for the first... The strategy output of the unmanned vessels other than the one unmanned vessel.

[0192] As a further preferred implementation, the above experience buffer adopts a uniform random sampling strategy. In each training step, sample data of a preset batch size are independently and randomly sampled from the global experience buffer and the local experience buffer, respectively, for parameter updates of the critic network and the actor network.

[0193] like Figure 4 As shown, this invention employs a multi-agent reinforcement learning framework with centralized training and distributed execution. During the training phase, two types of experience buffers are maintained: global and local, storing global state transition data and local observation data, respectively. The critic network completes value evaluation based on the global state and the joint actions of the entire formation, while the actor network independently generates control actions based solely on the local observations of each unmanned vessel, meeting the engineering requirements of distributed execution. The implementation of the policy sequential update mechanism in this framework includes: in each outer training loop, the actor networks of each unmanned vessel are updated sequentially according to a preset order. When updating the policy parameters of a single unmanned vessel, the policies of all other unmanned vessels remain fixed; only the actor and corresponding critic network parameters of the current agent are updated. This method eliminates the environmental non-stationarity caused by simultaneous updates of multiple agents at the source, significantly improving training stability and policy convergence reliability.

[0194] For step 7, output the trained formation collision avoidance strategy. After the training meets the preset convergence condition, save the actor network parameters and commentator network parameters of each unmanned vessel, and output the trained formation collision avoidance strategy.

[0195] Step 8, performing online formation collision avoidance of multiple unmanned vessels, is specifically divided into the following steps:

[0196] Step 8.1, Acquire Local Observations. Each unmanned surface vessel acquires local observations in real time. .

[0197] Step 8.2, output control action. This involves controlling the local observation. Input the trained actor network, output continuous motion. The reference heading guidance information has been integrated into the local observation, so the strategy is constrained by the collision avoidance direction during the action generation process, so that the output action is preferentially oriented towards the safe and feasible area; at the same time, for the follower, when the collision avoidance trigger condition is met, its speed is constrained to avoid path oscillation or invalid adjustment caused by excessively fast movement, thereby improving the stability and efficiency of the formation collision avoidance process.

[0198] For step 9, the online execution phase monitors the current leader's operational status in real time. If the leader loses its connection with the formation, it is determined that the current leader has failed; otherwise, step 8 continues.

[0199] For step 10, when the leader fails, the leader re-election mechanism is activated and the formation task continues. This is specifically divided into the following steps:

[0200] Step 10.1: Retain the original leader's task information, including the target list and the current target index.

[0201] Step 10.2, calculate the reselection distance of candidate followers. Let the original leader's current target point be... , No. The current position of the candidate follower is Then its reselection distance is defined as:

[0202]

[0203] in, It is an L2 norm.

[0204] Step 10.3, determine the new leader. The system selects a leader that meets the following criteria: Unmanned ships are emerging as a new leader, among which This represents the set of all unterminated followers. This represents the distance of each candidate follower to the original leader's objective point. The new leader inherits the original leader's mission information, and the remaining followers recalculate their respective objective points based on the new leader's position and heading, continuing to perform the formation mission.

[0205] Based on the above design of the embodiments of the present invention, the following provides more specific simulation examples to further demonstrate the implementation of the solution:

[0206] 1. Formation Simulation Environment Construction

[0207] 1.1 Environment Initialization

[0208] First, a two-dimensional water simulation environment is constructed, with the environmental range set as follows: ,in Several obstacles are randomly placed within this area, including stationary static obstacles and dynamic obstacles with a certain speed. The dynamic obstacles move according to preset rules at each moment, and their speed varies within a given range to simulate interference targets in a real marine environment.

[0209] 1.2 Formation Structure Establishment

[0210] Multiple unmanned surface vessels (USVs) are deployed in a formation within an environment, and the USV assembly is defined as... One of the unmanned vessels is randomly selected as the leader, and the rest are followers. The position and heading of each unmanned vessel are initialized, i.e., its initial state is set. and speed At the same time, set a global goal for the leader. , serving as the navigation target for the entire formation.

[0211] 1.3 Formation Target Generation

[0212] Based on the leader's current location and course, generate corresponding formation target points for each follower ship in real time. The target point represents the desired position of the follower within the formation. This is used for subsequent control and reward calculation.

[0213] 2. Formation Collision Avoidance Operation Procedure

[0214] After the system initialization is complete, each unmanned vessel proceeds according to the time step. The following process is executed repeatedly:

[0215] 2.1 Status Information Collection

[0216] At any moment For the first The unmanned vessel first collects current status information, including:

[0217] (1) Self-state With speed ;

[0218] (2) Target point information ;

[0219] (3) Detect obstacle information within the detection range;

[0220] (4) Information on nearby unmanned vessels.

[0221] The above information is combined to form a local observation. .

[0222] 2.2 Calculation of Key Variables

[0223] Based on the current observation information, calculate the following variables:

[0224] Deviation between current position and target point ;

[0225] Target direction ;

[0226] Distance of obstacles in all directions And obtain the minimum obstacle distance. .

[0227] 2.3 Dynamic Adjustment of Target Points

[0228] Based on minimum obstacle distance Determine whether the current state has entered collision avoidance mode. At the same time, maintain the original formation target point. ;when At that time, switch the target point to the leader's target point. Get the current target point. .

[0229] 2.4 Collision Avoidance Direction Calculation

[0230] when At that time, the collision avoidance guidance mechanism is activated in the direction set. Internally calculate the cost function in each direction. And select the optimal direction As a reference direction for collision avoidance; when At that time, the target direction is directly adopted. .

[0231] 2.5 Determining the Desired Course

[0232] Determine the desired course based on the current status. And calculate the heading deviation. As part of the decision input.

[0233] 2.6 Decision Input Construction

[0234] The following information is combined to form the decision input:

[0235] (1) Self-state information ;

[0236] (2) Obstacle Information ;

[0237] (3) Neighboring vessel information .

[0238] Obtain the local observation vector And input it into the actor network.

[0239] 2.7 Action Output and Execution

[0240] Actor network based on input Output Action and convert it into a control variable. and The unmanned vessel's state is updated based on the control input to obtain the state for the next moment. .

[0241] 2.8 Constraint Handling

[0242] When the unmanned vessel is a follower and satisfies At that time, a constraint is imposed on its speed to ensure that it meets the following conditions. .

[0243] 2.9 Reward Calculation

[0244] Calculate the reward based on the current status. This includes items related to formation error, heading deviation, obstacle safety distance, and intra-formation safety distance.

[0245] 2.10 Data Storage

[0246] Data at the current moment and Store it in the experience pool for subsequent training.

[0247] 2.11 Strategy Sequence Update

[0248] In each training cycle, data is sampled from the experience pool to update the policies of each unmanned surface vessel (USV). The update order is randomized by index. First, the first agent is selected, and its policy parameters are updated using the sampled data. Then, the second agent is selected, and its policy is updated while keeping the policy of the first agent fixed. This process continues until the next agent is selected. Each agent has completed the update.

[0249] 2.12 Leader Status Assessment

[0250] Determine if the current leader has become ineffective. If not, proceed to the next time step. Continue with steps 2.1 to 2.11; if this fails, proceed to step 2.12.

[0251] 2.13 Leader re-election

[0252] Calculate the distance from each follower to the leader's current target point. Select the unmanned vessel corresponding to the minimum value as the new leader and update the formation structure. After updating, return to step 2.1 to continue execution.

[0253] 3. Training termination conditions

[0254] The current turn ends when any of the following conditions are met:

[0255] (1) The leader arrives at the target point;

[0256] (2) All unmanned vessels collide or cross the boundary;

[0257] (3) Reach the maximum time step.

[0258] 4. Strategy Training and Application

[0259] During the training phase, the above steps are repeated to update the policy network until convergence. After training, the trained policy is used for online formation collision avoidance control, with each unmanned surface vessel relying only on local observations. Collaborative decision-making can then be completed.

[0260] As a further preferred implementation method, the convergence criteria for model training are as follows: within 40 consecutive evaluation rounds, the formation task completion rate is ≥95%, the collision avoidance success rate is ≥98%, and the cumulative reward fluctuation range is ≤5%. When the above conditions are met, the model is considered to have converged, training is stopped, and the network parameters are saved.

[0261] As a reference for preferred implementation, the unmanned surface vessel (USV) model parameters, collision avoidance parameters, reward parameters, and training parameters used in the above examples are shown in Tables 1 to 4. The parameters can be modified according to different USV models to meet different needs.

[0262] Table 1. Dynamics and Physical Model Parameters of Unmanned Vessels

[0263]

[0264] Table 2. Parameters related to the collision avoidance algorithm

[0265]

[0266] Table 3 Reward-related parameters

[0267]

[0268] Table 4 Training Hyperparameters

[0269]

[0270] Compared with existing technologies, the solutions proposed in the embodiments of the present invention can achieve targeted improvements in key issues of multi-unmanned vessel collaborative decision-making: by reducing environmental non-stationarity during multi-agent training through a policy order update mechanism, the stability of policy convergence and the reliability of collaborative decision-making are improved; by fusing vector field histogram guidance and collision avoidance constraint mechanisms, invalid actions in complex close-range interaction scenarios are reduced, collision avoidance efficiency is improved and formation operation stability is enhanced; and by employing a formation robustness mechanism oriented towards local observation and leader failure scenarios, a unified target reference and formation structure are maintained, improving the system's adaptability, task continuity and overall robustness in complex dynamic environments.

[0271] like Figure 7 As shown, existing methods in obstacle-dense formation collision avoidance scenarios are prone to ineffective actions such as frequent turns, significant track oscillations, and followers deviating severely from the desired formation position. This not only increases track redundancy and reduces collision avoidance efficiency but also leads to a scattered formation structure, insufficient multi-ship coordination, and even the risk of ship-to-ship collisions. The solution provided in this invention, through a VFH-guided collision avoidance constraint mechanism, imposes a safe interval constraint on the action search space of the reinforcement learning strategy. Combined with follower velocity constraints during collision avoidance, this effectively eliminates such ineffective actions, significantly improving the stability and execution efficiency of the formation collision avoidance process.

[0272] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0273] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0274] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0275] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0276] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive various other forms of a multi-agent reinforcement learning unmanned surface vessel formation collision avoidance method based on policy order updates. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A multi-agent reinforcement learning-based collision avoidance method for unmanned surface vessels (USVs) based on policy order updates, characterized in that: The model training phase includes: constructing a multi-UAV formation collaborative decision-making environment; establishing a leader-follower formation topology; designating one UAV as the leader to provide global navigation guidance, with the remaining UAVs acting as followers; generating the formation's desired target point based on the leader's real-time pose; constructing a local observation vector for each UAV in the formation, containing its own state information, relative obstacle position information, and neighboring vessel interaction information; and constructing a multi-agent reinforcement learning decision-making model based on the local observation vectors, including a policy network and a value evaluation network. The policy network outputs continuous control actions based on the local observation vectors of a single UAV, while the value evaluation network performs value evaluation based on the global state and the joint actions of the entire formation. To adapt the reward function for formation maintenance and cooperative collision avoidance, a collision avoidance guidance mechanism based on vector field histograms is introduced. This mechanism dynamically switches the desired course of the unmanned vessel (UV) based on its minimum obstacle distance, integrating the switched desired course into the local observation vector. Simultaneously, when the UV enters the collision avoidance state, the current task target point of the follower is switched from the formation's desired target point to the leader's global target point, thereby forming a safe direction constraint on the action selection of the policy network. A policy sequence update mechanism is used to train the multi-agent reinforcement learning decision model: in each training cycle, the model parameters of each UV are updated sequentially according to a preset order. When updating the model parameters of a single UV, the model parameters of all other UVs in the formation are fixed. The online execution phase includes: deploying the trained policy network to each unmanned vessel; when performing the formation collision avoidance task online, each unmanned vessel independently outputs control actions based on the real-time acquired local observation vectors; real-time detection of the formation leader's operating status; when the original leader is detected to have failed, calculating the distance between each candidate follower and the original leader's current global target point, selecting the follower with the smallest distance as the new leader, the new leader inherits the original leader's global task information, and the remaining followers reconstruct the formation's expected target point based on the new leader's real-time pose, and continue to perform the formation collision avoidance task; In the collision avoidance guidance mechanism based on vector field histograms, the specific rules for dynamically switching the desired course are as follows: when the minimum obstacle distance of the unmanned vessel is greater than the preset collision avoidance trigger threshold, the direction of the unmanned vessel's current mission target point is taken as the desired course; when the minimum obstacle distance is between the preset proximity safety threshold and the collision avoidance trigger threshold, the local collision avoidance reference course generated by the vector field histogram is taken as the desired course; when the minimum obstacle distance is less than or equal to the proximity safety threshold, the direction of the unmanned vessel's current mission target point is taken as the desired course; when the minimum obstacle distance of the unmanned vessel is less than the collision avoidance trigger threshold, speed constraints are applied to the followers in the formation to ensure that the real-time speed of the followers does not exceed the real-time speed of the current leader. The detection of the formation leader's operational status specifically involves real-time detection of the leader's communication connection status with the other unmanned vessels in the formation. When the leader loses its communication connection with the formation, the original leader is deemed to have failed. The new leader inherits the original leader's global mission information, including the global navigation target point list and the current mission execution progress. The self-state information in the local observation vector is used to characterize the role attributes, heading error, and current speed of the unmanned vessel. The heading error is the deviation between the current heading and the desired heading.

2. The multi-agent reinforcement learning unmanned ship formation collision avoidance method based on policy sequential update according to claim 1, characterized in that: In the strategy sequence update mechanism, the preset sequence is the update order generated by randomly sorting all unmanned vessels in the formation before the start of each training cycle. In each training cycle, the next training cycle will only begin after all unmanned vessels have completed one round of model parameter updates.

3. The multi-agent reinforcement learning unmanned surface vessel formation collision avoidance method based on policy order update as described in claim 1, characterized in that: The policy network extracts features from the relative position information of obstacles and the interaction information of neighboring ships in the local observation vector through the attention encoding module. The extracted features are concatenated with the network's own state information and then input into the multilayer perceptron to output control actions. The value evaluation network adopts a dual-commentator parallel structure. The two sub-networks with the same structure output independent action value evaluation results. When updating the model parameters, the minimum value of the two evaluation results is selected to participate in the target value calculation.

4. The multi-agent reinforcement learning unmanned surface vessel formation collision avoidance method based on policy order update as described in claim 1, characterized in that: The reward function for adaptive formation keeping and cooperative collision avoidance is a weighted composite reward function, which includes formation keeping reward, heading reward, obstacle safety distance reward, intra-formation safety distance reward and event-based reward and penalty. Among them, the formation keeping reward includes continuous keeping reward within the target tolerance range.

5. The multi-agent reinforcement learning-based unmanned surface vessel (USV) formation collision avoidance method based on policy order update as described in claim 1, characterized in that: When performing formation collision avoidance missions online, each unmanned vessel completes distributed collaborative decision-making based solely on its own local observation vectors, without requiring global state information or full connectivity communication with the entire formation.

6. A multi-agent reinforcement learning unmanned surface vessel (USV) convoy collision avoidance system, characterized in that, The system for performing the method of claim 1 includes a formation configuration module, a perception preprocessing module, a reinforcement learning decision module, a collision avoidance guidance module, a training control module, and an online execution module; The formation configuration module is used to construct a collaborative decision-making environment for multiple unmanned vessels, establish a leader-follower formation topology, and complete the role configuration of the leader and followers and the generation of formation objectives. The perception preprocessing module is used to collect the self-state of each unmanned vessel, the relative position of obstacles and the interaction information with neighboring vessels, and to construct local observation vectors. The reinforcement learning decision module includes a policy network unit and a value evaluation network unit, which are used to generate continuous control actions based on local observation vectors and complete the value evaluation during the intensive training phase. The collision avoidance guidance module is used to dynamically switch the desired course based on the minimum obstacle distance of the unmanned vessel according to the vector field histogram algorithm, integrate the desired course into the local observation vector to complete the action constraint, and complete the dynamic switching of the follower target point in the collision avoidance state. The training control module is used to execute the strategy order update mechanism and control the update order and update logic of the parameters of each unmanned surface vessel model. The online execution module is used to deploy the trained policy network and complete the distributed formation collision avoidance task execution, leader state detection, and leader re-selection operation.

7. A multi-agent reinforcement learning unmanned surface vessel formation collision avoidance system according to claim 6, characterized in that, The training control module has a built-in random sorting unit, which is used to randomly sort all unmanned vessels in the formation before the start of each training cycle to generate the model parameter update order; the collision avoidance guidance module has a built-in threshold control unit and speed constraint unit, which are used to execute the segment switching logic of the desired course and the speed constraint of the follower.

8. A multi-agent reinforcement learning unmanned surface vessel formation collision avoidance system according to claim 6, characterized in that, The online execution module has a built-in distributed decision-making unit, which enables each unmanned vessel to generate control actions independently based on its own local observation vectors, without the need for global state information interaction.

Images