Multi-unmanned surface vessel cooperative hunting method for multi-obstacle environment

Abstract

The invention discloses a multi-unmanned surface vessel cooperative hunting method for a multi-obstacle environment, and relates to the technical field of unmanned surface vessel cooperative control and multi-agent reinforcement learning, and the method comprises the steps: firstly constructing a cooperative hunting system model comprising a target hunting scene model, a USV kinematics model and a target escape strategy model; further designing an RA-MADDPG algorithm, wherein the core of the algorithm comprises a relationship awareness attention commentator (RAAC) network, a sparse proximity awareness observation model and a stage self-adaptive reward mechanism; after the algorithm is trained and deployed through an experience playback pool, the multiple USVs can efficiently and synergistically surround the target in a multi-obstacle environment. A simulation experiment shows that compared with a reference algorithm, the method has the advantages that the average reward is improved by about 36.5%, the surrounding success rate is improved by 17.1%, the task completion time is shortened by 17.4%, the collision rate is reduced by 77.8%, rapid, safe and stable cooperative surrounding in a complex environment is effectively achieved, and the method is suitable for scenes such as overwater security and protection, search and rescue and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned surface vehicle cooperative control and multi-agent reinforcement learning, and particularly relates to a multi-unmanned surface vehicle cooperative hunting method for a multi-obstacle environment. BACKGROUND

[0002] Unmanned surface vehicles (USVs) have been widely used in the fields of marine resource exploration, water security patrol, search and rescue, etc. due to their autonomous operation capability, continuous task execution capability and safety without direct human control. In particular, in dynamic target cooperative hunting tasks such as illegal ship interception and out-of-control floating object containment, multiple USVs can quickly form a containment situation through division of labor and cooperation, significantly improving the task success rate and response efficiency, and becoming a core technology direction in this field.

[0003] Multiple USV cooperative hunting tasks face multiple complex challenges: first, the target often has autonomous evasion capability, and the motion trajectory is uncertain, requiring the USV cluster to dynamically adjust the strategy to prevent the target from escaping; second, the working water area often has islands, reefs, other ships and other static or dynamic obstacles, and the USV needs to accurately avoid obstacles during the hunting process to ensure its safety; third, the cooperative coordination among multiple USVs is a core difficulty, and under the condition of limited information interaction, how to achieve efficient synchronization of task allocation, path planning and action coordination directly affects the completion quality of the hunting task.

[0004] To solve the above problems, related technologies have evolved from traditional model-based methods to reinforcement learning methods: I. Traditional cooperative hunting methods Early research mainly adopted preset behavior rules or mathematical modeling ideas, including: 1. Geometric and bionics methods: by simulating wolf hunting strategies, predator-prey systems and other biological cooperation mechanisms, or constructing bionic neural networks to achieve hunting; 2. Game theory methods: abstract the hunting as a pursuit and evasion game problem, and obtain the optimal strategy by analytical or numerical solution, such as using Zermelo-Voronoi region division, Voronoi-Apollonius hybrid strategy, etc.; 3. Cooperative control methods: achieve cooperative hunting without communication through cooperative control and distributed optimal control.

[0005] However, these traditional methods have significant limitations: they are heavily dependent on static environment assumptions and accurate system parameters, and have poor adaptability in the face of dynamic scenarios such as unknown targets, sudden evasion behaviors or agent failures, which can lead to a significant decline in hunting performance and make it difficult to meet the actual needs of complex water environment.

[0006] II. Reinforcement learning cooperative hunting method To overcome the defects of traditional methods, reinforcement learning, especially deep reinforcement learning, is widely used in multi-agent hunting tasks. Deep reinforcement learning combines the perception ability of deep learning and the decision optimization ability of reinforcement learning, enabling agents to learn strategies autonomously in complex partially observable environments.

[0007] In the multi-agent scenario, multi-agent reinforcement learning algorithms that support centralized training and distributed execution frameworks and are suitable for continuous action spaces have become a research hotspot, such as MADDPG, MAPPO, etc. Compared with algorithms such as COMA, VDN, QMIX, etc. for discrete action spaces, they are more suitable for the continuous control requirements of USV hunting. Researchers have made many optimizations for such algorithms, for example: role-based heterogeneous agent design combined with Voronoi reward mechanism to improve robustness in unknown obstacle environments; fusion of particle swarm optimization and M3DDPG to solve the problems of sample sparsity and unstable convergence; design of double reward mechanism to solve the problem of sparse rewards; introduction of adaptive environment generator to improve strategy generalization ability; design of phased guidance reward function based on MASAC to promote cooperation, etc.

[0008] Although existing reinforcement learning methods have made some progress, there are still core challenges in the multi-USV cooperative hunting task in a multi-obstacle environment: 1. Insufficient modeling of multi-agent interaction: The critic network of traditional algorithms (such as MADDPG) usually uses a simple concatenation method to fuse information from other agents, assuming that the interaction strength between all agents is homogeneous and static, which makes it difficult to capture the evolving cooperative relationship in complex dynamic environments, resulting in low Q value estimation accuracy and poor algorithm convergence efficiency; 2. Observation dimension redundancy problem: In a multi-obstacle environment, if all obstacle information is obtained, it may cause dimension explosion, and traditional observation models lack effective information filtering mechanisms, leading to increased computational burden and affecting real-time decision-making performance; 3. Inaccurate reward mechanism guidance: Existing reward functions have difficulty in balancing the relationship between hunting efficiency, obstacle avoidance safety, and target escape avoidance, especially lacking adaptive guidance for different stages of hunting, which cannot provide precise optimization direction for multi-agent cooperative behavior.

[0009] In summary, existing technologies still have problems such as low hunting efficiency, high collision risk, and poor cooperative consistency when dealing with the coupled challenges of dynamic targets, complex obstacle environments, and multi-agent cooperation. There is an urgent need for a technical solution that can optimize multi-agent interaction modeling, achieve efficient state representation, and provide precise stage-adaptive reward guidance to meet the practical needs of multi-USV cooperative hunting in complex water environments quickly, safely, and stably. SUMMARY

[0010] To address this, this invention provides a method for collaborative encirclement and capture of multiple unmanned surface vessels in multi-obstacle environments. This method solves the problems of low encirclement efficiency, high collision rate, and poor coordination consistency in existing technologies for collaborative encirclement and capture of multiple USVs. These problems stem from insufficient multi-agent interaction modeling, redundant observation dimensions in multi-obstacle environments, and the difficulty in accurately balancing encirclement efficiency, obstacle avoidance safety, and target escape avoidance through reward mechanisms.

[0011] To address the aforementioned technical problems, embodiments of the present invention provide a method for coordinated encirclement and capture of multiple unmanned surface vessels in environments with multiple obstacles, the method comprising the following steps: A multi-unmanned surface vessel (UAV) collaborative encirclement and capture system model is established. The system model includes a target encirclement and capture scenario model, an UAV kinematic model, and a target escape strategy model. The target encirclement and capture scenario model defines the geometric and distance conditions for successful encirclement and capture. The UAV kinematic model describes the changes in position, heading angle, linear velocity, and angular velocity of the UAV through differential equations. The target escape strategy model calculates the repulsive force between the encirclement vessel and obstacles on the target based on the artificial potential field method to determine the target's direction of motion. The RA-MADDPG algorithm is designed, which includes a relation-aware attentional commentator network, a sparse proximity-aware observation model, and a stage-adaptive reward mechanism. Based on the unmanned surface vessel cooperative encirclement system model and the RA-MADDPG algorithm, an experience replay pool is constructed and the algorithm is trained. The parameters of the relational awareness commentator network are updated by minimizing the temporal difference error, the parameters of the Actor network are updated by policy gradient, and the parameters of the target network are updated by soft update. The trained RA-MADDPG algorithm is deployed to multiple unmanned surface vessels to control them to perform cooperative encirclement and capture missions in multi-obstacle environments, achieving rapid encirclement and stable maintenance of the target.

[0012] Preferably, the construction process of the relation-aware attentional critic network includes: All observation vectors of unmanned surface vessels Corresponding actions By concatenating the data, the encoder input is obtained. ,in , For the dimension of the observation vector, For the action vector dimension; Will Projecting onto hidden layer features via fully connected layer ,in , This is the weight matrix. For bias terms, For hidden layer dimensions; set up Each independent attention head generates diverse feature representations. ,in For the first The weight matrix of each attention head. For the first Bias terms for each attention head; splicing the motion vectors of all unmanned surface vessels except the current one Encoded as decoder state ,in , This is the decoder weight matrix. For decoder bias terms; Calculate the interaction score between each attention head and the decoder state. Attention weights are obtained by normalization using the Softmax function. ; The context vector is obtained through attention weighted aggregation. ,Will Input two fully connected layers and output Value estimation.

[0013] In the sparse proximity sensing observation model, the observation space of each unmanned surface vessel From its own state of motion Teammate status Target state and the status of nearby obstacles Composition, that is ;in: The , , The coordinates of the unmanned surface vessel's position. The baseline length for scene normalization. The velocity direction angle, For velocity modulus, This is the maximum speed of the unmanned surface vessel; The It includes sine and cosine encoding of the relative position, relative velocity magnitude, and relative velocity direction between the unmanned surface vessel and its teammates; The It includes sine and cosine encoding of the relative position, relative velocity magnitude, and relative velocity direction between the unmanned surface vessel and the target; The ,in The relative position of the unmanned surface vessel to the nearest obstacle. The radius is the nearest obstacle, and is only true if the nearest obstacle is within the detection radius of the unmanned surface vessel. Non-zero, otherwise set to 0.

[0014] Preferably, the phase-adaptive reward mechanism is based on a three-phase division of "approach-encirclement-maintenance," and the phase transition is determined according to the relative positional relationship between the unmanned surface vessel and the target. When at least one unmanned surface vessel is at a distance from the target >Threshold radius of the encirclement phase At that time, it enters the approach phase; When all unmanned surface vessels are at a distance from the target ≤ Encirclement stage threshold radius If the conditions for encirclement are not met, the process enters the encirclement phase, in which... Expected capture radius for successful encirclement ; When all unmanned surface vessels simultaneously meet the conditions for encirclement, the process enters the maintenance phase. The conditions for the encirclement and capture are as follows: and , Unmanned surface vessel With adjacent unmanned surface vessels The relative azimuth angle, , The number of unmanned surface vessels. To account for the allowable angle error, This represents the allowable distance error.

[0015] Preferably, the reward function in the stage adaptive reward mechanism ,in This is a global penalty item. The rewards are awarded in stages. The ; The As a collision penalty, when the unmanned surface vessel is close to an obstacle... ≤ obstacle safety distance hour, When the distance between unmanned surface vessels ≤Safe distance hour, ,in , The penalty coefficient is... To prevent division by zero constant; The As a time penalty, ,in For the current time step, For the maximum time step, This represents the time penalty coefficient.

[0016] Preferably, the phased reward items The expression in the encirclement phase is: ; The ,in This is the distance maintenance coefficient, and It is approximately 1 / 3 of the distance coefficient for the approach stage. Indicates the distance between the unmanned surface vessel and the target. The desired capture radius; The ,in Let V be the variance of the distances from all unmanned surface vessels to the target. This is the distance balance coefficient. This refers to the number of unmanned surface vessels; The ,in Unmanned surface vessel Maximum angular deviation, The angular uniformity coefficient; The To align rewards, when and hour, Otherwise, it is 0, where For distance tolerance, This represents the allowable error for the angle.

[0017] Preferably, the phased reward items The expression during the maintenance phase is: ; The To maintain a fixed reward, ; The As a reward for success, the time the unmanned surface vessel maintains the encirclement state will be counted. hour, Otherwise, it is 0.

[0018] Preferably, the differential equation of the kinematic model of the unmanned surface vessel is: ; in, This represents the position coordinates of the unmanned surface vessel in a two-dimensional plane. The heading angle is defined as the angle between the longitudinal axis of the hull and the positive x-axis of the global coordinate system. The linear velocity along the heading; Angular velocity; This is the actual linear acceleration command; This is the actual angular acceleration command. For maximum linear acceleration, This is the maximum angular acceleration.

[0019] Preferably, in the target escape strategy model, the total repulsive force experienced by the target is... ; in , , For the target location, Unmanned surface vessel Location, Unmanned surface vessel Distance to the target Obstacles Location, Obstacles Distance to the target The number of unmanned surface vessels. The number of obstacles, , It is the repulsion coefficient; when At that time, the target's instantaneous velocity ,when At that time, the target maintains its original direction of motion, where This is the maximum linear acceleration.

[0020] Preferably, the process of constructing an experience replay pool and training the algorithm based on the unmanned surface vessel cooperative encirclement system model and the RA-MADDPG algorithm, updating the parameters of the relation-aware attention commentator network by minimizing the temporal difference error, updating the parameters of the Actor network by policy gradient, and updating the parameters of the target network by soft update includes: Relationship-aware attention critic network minimizes the loss function of temporal difference error. Update parameters, where For the first Relationship perception and attention commentator network parameters of an unmanned surface vessel; For the expectation operator on the experience replay pool, This is a local collection of observations from all unmanned surface vessels at the current moment. This is the set of actions of all unmanned surface vessels at the current moment. , For the current moment, the first Instant rewards for unmanned surface vessels This is a localized collection of observations from all unmanned surface vessels at the next moment. For experience replay pool; For the first The target of the unmanned surface vessel value, , For the first Instant rewards for unmanned surface vessels As a discount factor, For the first A network of commentators on the target relationship perception and attention of unmanned surface vessels. Value estimation, For target relationship-aware attention commentator network parameters, This is the set of actions of all unmanned surface vessels for the next moment; For the first The current relational perception attention of an unmanned surface vessel (USV) - commentator network Value estimation; Actor networks utilize policy gradients. Update parameters, where For the first The policy gradient of an unmanned surface vessel in the Actor network. In order to address the "first" The gradient operator for calculating the partial derivatives of the Actor network parameters of an unmanned surface vessel. For the long-term expected return of the Actor network; For the observation expectation operator on the experience replay pool; For the first The gradient of the policy function of an unmanned surface vessel with respect to its own parameters. For the first Actor network policy function for an unmanned surface vessel For the first Local observations of an unmanned surface vessel; For the first The relationship between unmanned surface vessels, perception of attention, commentators, network The gradient of the value with respect to its own action. For the first The relationship between unmanned surface vessels, perception of attention, commentators, network Value functions; For the first The actions of an unmanned surface vessel; The target network parameters are updated via soft updates: ,in This is the soft update coefficient. For the current relation-aware attention commentator network parameters, For target relationship-aware attention commentator network parameters, These are the current Actor network parameters. The parameters of the target Actor network.

[0021] As can be seen from the above technical solutions, this invention application has the following beneficial effects: (1) This invention, by designing a Relationship-Aware Attention Critics (RAAC) network, overcomes the limitations of simple concatenation of agent information in traditional algorithms. It adaptively adjusts the interaction intensity between agents based on the motion state of each USV and the local environment, accurately models dynamic collaborative relationships, and significantly optimizes Q-value estimation accuracy and algorithm convergence efficiency. This enables efficient action coordination and policy synchronization among multiple USVs during the encirclement process, effectively shortening task completion time. Simulation results show that compared to the benchmark algorithm MADDPG, this method increases the average reward by approximately 36.5%, shortens the task completion time by 17.4%, and substantially improves encirclement efficiency.

[0022] (2) The sparse proximity perception observation model proposed in this invention, by filtering the key state information of the nearest obstacle and using relative position encoding, achieves fixed-dimensional state representation while avoiding dimensional explosion and computational redundancy caused by full obstacle information, significantly reducing the decision-making computation burden. Combined with the progressive collision penalty design based on the inverse proportional function in the reward mechanism, the USV can accurately perceive dangerous areas and avoid them in advance. Experimental verification shows that in a complex scenario with 10 obstacles, the collision rate of this method is only 4%, which is 77.8% lower than that of MADDPG. While ensuring the continuous operation of the encirclement task, it greatly improves the operational safety and environmental adaptability of the USV itself.

[0023] (3) The phase-adaptive reward mechanism of this invention is based on a three-stage dynamic division of "approach-encirclement-maintenance" and a dual-radius buffer strategy, providing precise guidance for the core targets in different encirclement stages: the approach stage focuses on rapidly shortening the distance, the encirclement stage emphasizes the uniformity of spatial distribution, and the maintenance stage ensures the stability of the encirclement state. This progressive guidance mode avoids strategy imbalance during the encirclement process and effectively addresses the target's autonomous avoidance behavior. Data shows that the encirclement success rate of this method reaches 95.9%, which is 17.1% higher than that of MADDPG, and it can stably maintain the encirclement state, significantly reducing the risk of target escape, and providing reliable protection for collaborative encirclement tasks in complex dynamic scenarios. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Referring to the drawings will make the features and advantages of the present invention clearer. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart of a multi-unmanned surface vessel collaborative encirclement and capture method for multi-obstacle environments provided by the present invention; Figure 2This is a schematic diagram illustrating the conditions for successful capture in this invention; Figure 3 This is a schematic diagram of the motion model of the unmanned surface vessel in this invention; Figure 4 This is a schematic diagram of the RA-MADDPG algorithm framework in this invention; Figure 5 This is a structural diagram of the RAAC in this invention; Figure 6 This is a schematic diagram of the three-stage task in this invention; Figure 7 This is a schematic diagram of the angle deviation in this invention; Figure 8 This is a schematic diagram of the RA-MADDPG algorithm in this invention; Figure 9 This is a schematic diagram of the initial agent distribution scenario in this invention; Figure 10 This is a schematic diagram illustrating the average reward during training for the four methods in this invention; Figure 11 This is a schematic diagram illustrating the average reward of the four methods when the number of obstacles increases from 0 to 10 in this invention; Figure 12 This is a schematic diagram illustrating the success rates of four methods when the number of obstacles increases from 0 to 10 in this invention. Figure 13 This is a schematic diagram of the encirclement time of four methods when the number of obstacles increases from 0 to 10 in this invention; Figure 14 This is a schematic diagram of the collision rate of the four methods in this invention when the number of obstacles increases from 0 to 10; Figure 15 This is a schematic diagram of the enclosed trajectory in this invention, where (a) is the MADDPG method; (b) is the SAC method; (c) is the MATD3 method; and (d) is the RA-MADDPG method. Figure 16 This is a schematic diagram of the distance characteristics of the pursuit boat changing over time in this invention, where (a) is the MADDPG method; (b) is the SAC method; (c) is the MATD3 method; and (d) is the RA-MADDPG method. Figure 17 This is a schematic diagram of the speed characteristics of the pursuit boat changing over time in this invention, where (a) is the MADDPG method; (b) is the SAC method; (c) is the MATD3 method; and (d) is the RA-MADDPG method. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1: To address the problems of low capture efficiency, high collision rate, and poor coordination in existing multi-USV cooperative capture techniques, such as insufficient multi-agent interaction modeling (e.g., traditional algorithm critic networks simply splice agent information, failing to capture dynamic cooperative relationships), redundant observation dimensions in multi-obstacle environments (easily leading to dimensional explosion and increased computational burden), and the difficulty in accurately balancing capture efficiency, obstacle avoidance safety, and target escape avoidance. Figure 1 As shown, this invention proposes a method for coordinated encirclement and capture of multiple unmanned surface vessels in environments with multiple obstacles. The method includes the following steps: S1: Establish a multi-unmanned surface vessel collaborative encirclement system model. The system model includes a target encirclement scenario model, an unmanned surface vessel kinematic model, and a target escape strategy model. Among them, the target encirclement scenario model defines the geometric and distance conditions for successful encirclement, the unmanned surface vessel kinematic model describes the changes in position, heading angle, linear velocity, and angular velocity of the unmanned surface vessel through differential equations, and the target escape strategy model calculates the repulsive force on the target from the encirclement vessel and obstacles based on the artificial potential field method to determine the target's direction of motion. S2: Design the RA-MADDPG algorithm, which includes a relation-aware attentional commentator network, a sparse proximity-aware observation model, and a stage-adaptive reward mechanism; S3: Based on the unmanned surface vessel cooperative encirclement system model and the RA-MADDPG algorithm, an experience replay pool is constructed and the algorithm is trained. The parameters of the relation-aware attention commentator network are updated by minimizing the temporal difference error, the parameters of the Actor network are updated by policy gradient, and the parameters of the target network are updated by soft update. S4: Deploy the trained RA-MADDPG algorithm to multiple unmanned surface vessels, control the multiple unmanned surface vessels to perform cooperative encirclement tasks in a multi-obstacle environment, and achieve rapid encirclement and stable maintenance of the target.

[0027] This invention proposes a collaborative encirclement method for multiple unmanned surface vessels (USVs) in multi-obstacle environments. It achieves efficient, safe, and stable collaborative encirclement through a four-step core process: First, step S1 establishes a collaborative encirclement system model, including a target encirclement scenario model (defining the geometric and distance conditions for successful encirclement), an USV kinematic model (precisely describing the USV's position, heading angle, and other motion states through differential equations), and a target escape strategy model (calculating the repulsive force on the target based on the artificial potential field method to determine its direction of motion). This provides a precise modeling foundation for subsequent algorithms, closely reflecting the actual scenario, and clarifies the encirclement judgment criteria and target behavior patterns. Next, step S2 designs the RA-MADDPG algorithm, which integrates a Relationship-Aware Attention Critics (RAAC) network, a sparse proximity-aware observation model, and a stage-adaptive reward mechanism. The RAAC network can adaptively adjust the interaction intensity between agents to optimize Q-value estimation, and the sparse proximity-aware observation... The test model can achieve efficient state representation with fixed dimensions to reduce computational redundancy. The phase-adaptive reward mechanism can accurately guide collaborative behavior in three stages: "approach-encircle-maintain," effectively solving the core pain points of insufficient interactive modeling, redundant observation dimensions, and inaccurate reward guidance in traditional algorithms. Then, step S3 is executed to build an experience replay pool based on the above system model and algorithm. By minimizing the temporal difference error to update the RAAC network parameters, updating the Actor network parameters through policy gradient, and updating the target network parameters through soft update, the algorithm converges stably and forms the optimal encirclement strategy, providing reliable model support for actual deployment. Finally, step S4 is executed to deploy the trained RA-MADDPG algorithm to multiple USVs to achieve rapid encirclement and stable maintenance of targets in multi-obstacle environments. Ultimately, the core objectives of improved encirclement efficiency, reduced collision rate, and enhanced anti-target escape capability are achieved, meeting the needs of actual scenarios such as water security and search and rescue.

[0028] In step S1, a model of a multi-unmanned surface vessel collaborative encirclement and capture system is established.

[0029] In a vast marine environment containing static obstacles, multiple isomorphic unmanned surface vessels (USVs) conduct patrol missions. Upon detecting an intruding vessel (target), the USV formation must coordinate to encircle it while avoiding obstacles. Geometrically, forming an effective encirclement of a target in two-dimensional space requires at least three agents forming a stable triangular formation to prevent the target from escaping in any direction. Therefore, this invention sets three USVs as the minimum operational unit for effective encirclement. Experiments will focus on this basic scenario to evaluate core collaborative capabilities, laying the foundation for future expansion to more complex scenarios.

[0030] 1. Target Encirclement Scenario Model Consider a simplified 2D bounded scene, containing A pursuit boat and an escaping target. The pursuit boat is typically marked in blue, and the escaping target in red. Successful encirclement requires specific geometric and distance conditions to be met: ; ; in, unmanned surface vessels The unmanned surface vessel next to it The relative azimuth angle between them It is the expected surrounding angle. This is the permissible angle error; unmanned surface vessels With the goal The distance between them It is the expected capture radius. This is the allowable distance error. A schematic diagram of the successful enclosure condition is shown below. Figure 2 As shown.

[0031] 2. Kinematic Model of Unmanned Surface Vessel

[0032] To focus on the three core conflicts of pursuit efficiency, obstacle avoidance safety, and cooperative coordination in multi-USV cooperative encirclement, and to simplify the motion planning problem, this invention adopts the following key assumptions: dynamic obstacles are not considered; the influence of hydrodynamic disturbances (such as ocean currents and waves) is ignored; the motion of USVs is constrained to a two-dimensional plane, ignoring the motion in the roll, pitch, and heave directions.

[0033] Based on the above assumptions, a simplified kinematic model is used to describe the motion state of the USV, which is described by the following differential equation: ; In the formula: This represents the position coordinates of the unmanned surface vessel in a two-dimensional plane. The heading angle is defined as the angle between the longitudinal axis of the hull and the positive x-axis of the global coordinate system. The linear velocity along the heading; Angular velocity; Linear acceleration command; This is an angular acceleration command. For maximum linear acceleration, This represents the maximum angular acceleration. A schematic diagram of the motion model is shown below. Figure 3 As shown.

[0034] 3. Target Escape Strategy Model The target's escape strategy employs a typical Artificial Potential Field (APF) framework. Specifically, each encirclement vessel and obstacle in the environment is considered a source of repulsive force at the target's location. A repulsive force is generated at the location: ; ; in, For the target location, Unmanned surface vessel (fishing vessel) Location, Unmanned surface vessel (fishing vessel) Distance to the target Obstacles Location, Obstacles Distance to the target The number of unmanned surface vessels. The number of obstacles, , Let be the repulsive force coefficient. The total virtual repulsive force on the target is the vector sum of the two: ; The direction of the target's instantaneous velocity is determined by the direction of the total repulsive field: ; when At that time, the target maintains its original direction of motion.

[0035] In step S2, the RA-MADDPG algorithm for cooperative target encirclement is designed. 1. Formalizing the problem The target acquisition mission of unmanned surface vessels (USVs) can be described as a partially observable Markov decision process (POMDP) ​​

[33] , which consists of tuples Definition. Wherein, Representing the state space, It is an action space; this invention employs a continuous action space. , This is the state transition function. For the reward function, For the strategy space, The observation space represents the independent local observations of each agent. . This is the discount factor. Reinforcement learning trains the optimal policy. To maximize expected cumulative discount rewards .

[0036] 2. RA-MADDPG Algorithm Framework As mentioned earlier, inefficient multi-agent information interaction is a key issue leading to policy inconsistency and bottlenecks in collaborative performance. The critic network in the traditional MADDPG algorithm typically uses a simple concatenation operation to fuse information from other agents. This method implicitly assumes that the interaction strength between all agents is homogeneous and static, making it difficult to effectively capture the continuously evolving collaborative relationships in complex dynamic environments. This limitation restricts the accuracy of value function estimation and ultimately affects policy consistency.

[0037] To address this fundamental problem, inspired by multi-head attention mechanisms, this invention proposes a Relation-Aware Attention Critic (RAAC) network, and names the enhanced algorithm Relation-Aware Attention Critic (RA-MADDPG). Compared to traditional concatenation-based critic networks, the RAAC network in RA-MADDPG explicitly models the dynamic interaction relationships between agents through an innovative global attention architecture: 1) it uses multi-branch feature representations to capture diverse interaction patterns; 2) it calculates attention weights through feature matching scoring functions; and 3) it achieves dynamic weighted aggregation of multi-head information. This design significantly enhances the value function's ability to represent complex cooperative and competitive relationships, thus laying a structural foundation for improving policy consistency and collaborative efficiency.

[0038] Algorithm framework such as Figure 4 As shown. In this multi-agent system, each agent has its own independent actor and critic network. At any given moment, each intelligent agent acquires its own observational information through the environment. It obtains its own action strategy through the actor network. After interacting with the environment, it obtains observation information for the next time step. With rewards This process is repeated. During this cycle, the current observation information, actions, rewards, and next observation information of all agents are collected. Store the samples in the experience replay pool. After training, randomly sample a portion of the samples from the experience replay pool for further training.

[0039] The core of the RAAC network architecture lies in explicitly modeling the dynamic interaction relationships between agents through a multi-head attention mechanism. Its architecture is as follows: Figure 5 As shown, the workflow of this network is as follows: First, collect the observation vectors of all agents. With action The encoder input is formed by splicing the components together. ; in, For the dimension of the observation vector, For the action vector dimension; This input is then projected onto the hidden layer features through a fully connected layer: ; in, , This is the weight matrix. For bias terms, For hidden layer dimensions.

[0040] Subsequently, K independent attention heads generate diverse feature representations in the following manner. Experiments have shown that optimal performance is achieved when the number of attention heads K=3: ; in, For the first The weight matrix of each attention head. For the first The bias term of attention head.

[0041] Meanwhile, the action splicing vectors of other agents Encoded as decoder state: ; in, , This is the decoder weight matrix. This is the decoder bias term.

[0042] The relationship-aware mechanism is implemented by dynamically calculating interaction weights: First, the interaction score between each attention head and the decoder state is calculated. ; The scores are normalized using the Softmax function to obtain the attention weights. ; Finally, a context vector of multi-view interaction information is synthesized. It is given by the following formula: ; This architecture retains the dynamic weighting properties of standard attention while improving the efficiency of multi-agent relationship modeling through a feature matching function. Context vectors Then it is processed through two fully connected layers, and finally output Value estimation.

[0043] 3. State Space In dynamic coordinated pursuit missions, the observation space of each pursuit vessel It consists of four parts, namely its own motion state. Teammate status Target state and the status of nearby obstacles The observation information is acquired through local sensor and communication interactions, and after sparsity filtering and normalization, a fixed-dimensional feature vector is formed. The specific design is as follows: .

[0044] 3.1 Self-motion state The state of the pursuit vessel includes its position, velocity direction, and velocity magnitude, defined as follows: ; ; in, , The coordinates are the location coordinates of the unmanned surface vessel (fishing boat). The baseline length for scene normalization. For the velocity direction angle, sine and cosine encoding is used to avoid periodic jumps in the angle. For velocity modulus, This is the maximum speed of the unmanned surface vessel (fishing boat).

[0045] 3.2 Teammate Status In a three-player coordinated pursuit scenario, each agent contains the following information about its two teammates: ; in, Indicates from the fishing boat Pointing to the capture boat The relative position vector, For the encirclement boat and fishing boat The direction of relative velocity, For the encirclement boat and fishing boat The relative velocity modulus, yes Components in the x and y directions.

[0046] 3.3 Target State The target's status is similar to that of their teammates: ; in, Indicates from the fishing boat Pointing to target The relative position vector, For the encirclement boat and target The direction of relative velocity, = For the encirclement boat and target The relative velocity modulus, yes The amount.

[0047] 3.4 Status of Nearby Obstacles In obstacle-dense environments, displaying all obstacles on the map would lead to dimensionality explosion and information redundancy. Therefore, this method stores only the state information of the nearest obstacle, reducing dimensionality while eliminating the limitation on the number of obstacles, making it more adaptable to dense obstacle environments. The obstacle information is defined as follows: ; in, Indicates the nearest obstacles and the encirclement boat. The relative positions between them yes The amount, This is the radius of the obstacle. At the same time, to prevent information about distant obstacles from misleading the pursuit vessel, a perception radius is set for the pursuit vessel. Only obstacles within the pursuit vessel's perception range will have their information displayed; otherwise, the obstacle information is set to 0.

[0048] 4. Movement space Based on the aforementioned state representation, the action space is designed to generate effective pursuit strategies. To adapt to the dynamic control requirements in multi-agent pursuit scenarios, this invention constructs a normalized action space based on decoupled kinematic parameters. For each unmanned surface vessel, its action space contains two independent continuous control dimensions: linear acceleration control input... With angular acceleration control input Its mathematical expression is: ; in, and Normalization to interval The continuous control signals represent the normalized longitudinal acceleration command and normalized heading angle acceleration command of the unmanned surface vessel at the current moment. These normalized signals are mapped into actual instantaneous control commands through physical system constraints: ; ; in, and These are the actual applied longitudinal and angular acceleration commands, respectively; and The maximum linear acceleration of the propulsion system and the maximum permissible angular acceleration of the actuators are defined to limit the physical feasibility of the commands. These commands are ultimately integrated through the kinematic model to drive the speed and heading status updates of the unmanned surface vessel.

[0049] 5. Reward Function In multi-agent cooperative encirclement tasks, the design of the reward function is crucial for guiding agents to learn effective cooperative strategies. This invention proposes a multi-dimensional reward mechanism based on stage partitioning, which solves the complex problem of cooperative encirclement of a moving target by encirclement boats through a progressive guidance strategy. Reward Function Global penalty item and phased reward items Composition, its mathematical expression is: .

[0050] 5.1 Global Penalty Items To ensure the safety of the agent's behavior and the efficiency of its learning, a global penalty term was designed that runs through all stages: ; Among them, collision penalty This is the core mechanism for ensuring the safety of multi-agent systems. The penalty term includes penalties between obstacles and agents, as well as penalties between agents themselves. It employs an inverse proportional function design, generating progressive penalties as agents approach dangerous areas. ; in, and The penalty coefficient is... This indicates the safe distance between the agent and the obstacle. Indicates the safe distance between intelligent agents. It is a very small constant that prevents division by zero errors. This design ensures that when an agent approaches an obstacle or other agent, the penalty intensity increases sharply with decreasing distance, effectively preventing collision behavior.

[0051] Time penalty To improve learning efficiency and encourage agents to complete tasks within a limited number of steps: ; in, Indicates the current time step. This represents the maximum time step allowed by the environment. This linear penalty term gradually increases as the task progresses, prompting the agent to optimize its action strategy to improve task efficiency.

[0052] 5.2 Stage Division Mechanism and Reward Function To effectively guide multi-agent learning of cooperative encirclement strategies, we propose a three-stage partitioning mechanism based on spatial distribution. Stage transitions are dynamically determined based on the relative positional relationship between the encirclement vessel and the target. ; in, To define the threshold radius for the enclosing stage, through The buffer mechanism prevents the encirclement boats from entering the high-precision control stage too early before a basic encirclement is formed. Figure 6 The three-stage task determination was demonstrated.

[0053] During the approach phase, the pursuit vessel is relatively far from the target, and its main objective is to shorten the relative distance to the target through distance guidance strategies. ; This reward uses an absolute value penalty mechanism; when the pursuit boat is far from the target position... The farther away, the greater the punishment, guiding the pursuit boat to quickly approach the target location.

[0054] When all the encirclement boats enter the threshold radius Inside, the system enters the encirclement phase. The core objective of this phase is to form a spatially uniform encirclement: ; During the encirclement phase, the pursuit boat still needs to approach the target, but the reward for getting closer is not dominant. Therefore, the coefficient for this phase is reduced to 1 / 3 of that for phase 1 to avoid interfering with the achievement of the main encirclement objective. .

[0055] Distance balance reward calculation: variance of the distance from all capture boats to the target. The smaller the variance, the greater the reward, which encourages the pursuit boat to maintain a constant distance from the target. .

[0056] To evaluate the uniformity of the encirclement, we designed an angle uniformity reward and calculated the maximum angular deviation of each encirclement boat to optimize the azimuth distribution of the boats around the target: ; ; in, This is the maximum angular deviation of the encirclement boat. This indicates the angular deviation of the right neighbor, that is, the absolute deviation between the actual angular distance between the current encirclement vessel and its right neighbor and the ideal angular distance, such as... Figure 7 As shown.

[0057] Furthermore, when a single encirclement vessel achieves the encirclement condition, this stage will reward the vessel with a certain amount of reward to encourage it to maintain this state: .

[0058] When all the encirclement vessels simultaneously meet the distance and angle conditions, the system enters the maintenance phase, the main objective of which is to maintain a stable encirclement: ; ; At this stage, the distance balance reward and angle uniformity reward still apply. Additionally, a fixed reward is given for each step. This compensates for the high-frequency control costs required to maintain precise positioning. A substantial completion reward is triggered when five consecutive steps are achieved, incentivizing the formation of a stable encirclement state.

[0059] In step S3, the algorithm is trained.

[0060] During the intensive training phase, the RAAC network is updated by minimizing the loss function of the temporal difference error: ; Where the target value The calculation is as follows: ; in, For the first Relationship perception and attention commentator network parameters of an unmanned surface vessel; For the expectation operator on the experience replay pool, This is a local collection of observations from all unmanned surface vessels at the current moment. This is the set of actions of all unmanned surface vessels at the current moment. , For the current moment, the first Instant rewards for unmanned surface vessels This is a localized collection of observations from all unmanned surface vessels at the next moment. For experience replay pool; For the first The target of the unmanned surface vessel value, For the first Instant rewards for unmanned surface vessels As a discount factor, For the first A network of commentators on the target relationship perception and attention of unmanned surface vessels. Value estimation, For target relationship-aware attention commentator network parameters, This is the set of actions of all unmanned surface vessels for the next moment; For the first The current relational perception attention of an unmanned surface vessel (USV) - commentator network Value estimation.

[0061] Actor network updates maximize expected return through policy gradient: ; in, For the first The policy gradient of an unmanned surface vessel in the Actor network. In order to address the "first" The gradient operator for calculating the partial derivatives of the Actor network parameters of an unmanned surface vessel. For the long-term expected return of the Actor network; For the observation expectation operator on the experience replay pool; For the first The gradient of the policy function of an unmanned surface vessel with respect to its own parameters. For the first Actor network policy function for an unmanned surface vessel For the first Local observations of an unmanned surface vessel; For the first The relationship between unmanned surface vessels, perception of attention, commentators, network The gradient of the value with respect to its own action. For the first The relationship between unmanned surface vessels, perception of attention, commentators, network Value functions; For the first The movement of an unmanned surface vessel.

[0062] The parameters of the target policy network and the target value network are updated via soft updates: ; in, This is the soft update coefficient. For the current relation-aware attention commentator network parameters, For target relationship-aware attention commentator network parameters, These are the current Actor network parameters. Let these be the parameters of the target Actor network. The above algorithms are summarized as Algorithm 1, such as... Figure 8 As shown.

[0063] In step S4, the trained RA-MADDPG algorithm is deployed to multiple unmanned surface vessels, which are then controlled to perform cooperative encirclement and capture missions in a multi-obstacle environment, achieving rapid encirclement and stable maintenance of the target.

[0064] To further illustrate the advantages of this invention, specific simulation experiments will be conducted below.

[0065] We constructed a simulation environment to verify the superior performance of the proposed algorithm. We compared the proposed method with three classic reinforcement learning algorithms: MADDPG, MATD3, and SAC, conducting a comprehensive evaluation based on five core metrics: average reward, encirclement success rate, encirclement time, collision rate, and performance visualization (such as trajectory plots). Subsequent sections will detail the environmental parameter settings and hyperparameter configurations, and provide a comprehensive analysis of the experimental results.

[0066] I. Environment Setup In this experiment, we constructed a simulation scenario of unmanned surface vessels surrounding a 6×6 meter water area. Obstacles with radii between [0.2, 0.3] meters are randomly distributed within this area. Three pursuit vessels are initially randomly distributed in the southern region, with coordinates ranging from [0.2, 0.3]. The target was initially located in the area north of the pursuit vessel, with coordinates ranging from [missing information]. All agents initially have a heading angle of 90°. The pursuit boat has a maximum speed of 0.2 m / s and an acceleration of 0.05 m / s²; the target's maximum speed is 0.12 m / s. This initial configuration ensures that the pursuit operation takes place in areas with dense obstacles: placing the target in the center would cause it to escape to open water prematurely, thus failing to effectively verify the algorithm's core performance in obstacle avoidance and cooperative encirclement. The initial agent distribution scenario is as follows: Figure 9 As shown.

[0067] The algorithms compared in this invention use uniform parameters, as shown in Table 1.

[0068] Table 1 Comparison of algorithm hyperparameter settings

[0069] II. Experimental Results 1. Average reward Average reward is a key metric for evaluating algorithm performance. This invention uses a sliding window average reward to evaluate algorithm performance, with the window size set to 100 episodes. Each episode records the cumulative total reward of the three agents. The window dynamically maintains the total reward data of the most recent 100 episodes. By calculating the average of these rewards, a smooth evaluation metric is obtained, which effectively reflects the convergence trend and learning efficiency of the algorithm. Figure 10 The average reward convergence of different algorithms during training is shown. It can be observed that the RA-MADDPG algorithm converges significantly faster than the other algorithms and achieves a higher average reward in a shorter time. This indicates that the RA-MADDPG algorithm can learn and adapt to the environment more quickly during training, thus improving learning efficiency.

[0070] Figure 11The average reward of each algorithm is shown under different numbers of obstacles. As can be seen from the figure, the RA-MADDPG algorithm maintains a high average reward across all obstacle numbers, demonstrating its robustness and superiority in different environments. The advantage of RA-MADDPG is particularly pronounced when there are many obstacles. With 10 obstacles, the average reward of RA-MADDPG is 361.9, an improvement of approximately 36.5% compared to the baseline MADDPG's 265.2, approximately 24.3% compared to MASAC's 291.1, and approximately 31.8% compared to MATD3's 274.6.

[0071] In summary, the RA-MADDPG algorithm not only exhibits faster convergence speed and higher average reward during training, but also demonstrates superior performance in evaluations with varying numbers of obstacles. This is mainly attributed to its explicit modeling of the interaction relationships between agents through the RAAC module, which enhances the Critic network's ability to process observation information and agent joint actions.

[0072] 2. Success rate of encirclement The success rate of encirclement refers to the percentage of missions in which the pursuit boat simultaneously meets both of the following conditions within a specified number of time steps: a. No collisions occurred: There were no collisions between the pursuit boats, between the pursuit boats and the target, or between the pursuit boats and obstacles; b. Successful encirclement and capture: The pursuit boat satisfied the requirements. and The conditions for encirclement and capture.

[0073] The evaluation was conducted by grouping 100 episodes into 10 independent tests, and the average of these tests was taken as the final result. Figure 12 As shown in the figure, the RA-MADDPG algorithm demonstrates excellent encirclement success rate under different numbers of obstacles. It can be seen from the figure that as the number of obstacles increases, the success rates of other algorithms show a downward trend, while the success rate of the RA-MADDPG algorithm, although also decreasing, remains at a high level and is significantly better than other algorithms.

[0074] Specifically, when the number of obstacles is 0, the success rate of all algorithms is close to 100%. However, as the number of obstacles increases, the success rates of the MADDPG and MATD3 algorithms drop significantly, especially when the number of obstacles reaches 6 or more, where the success rate falls below 90%. In comparison, the MASAC algorithm performs slightly better, but its success rate also shows a downward trend as the number of obstacles increases.

[0075] With 10 obstacles, the RA-MADDPG algorithm achieved a success rate of 95.9%, which is approximately 17.1% higher than the baseline MADDPG (81.9%), approximately 9.4% higher than MASAC (87.7%), and approximately 13.5% higher than MATD3 (84.5%). This further demonstrates the superior performance of the RA-MADDPG algorithm in complex environments.

[0076] While the success rate of the RA-MADDPG algorithm decreases slightly with an increased number of obstacles, the decrease is small and it consistently remains above 95%, demonstrating its robustness and superiority in complex environments. This indicates that the RA-MADDPG algorithm exhibits better adaptability and stability when handling encirclement tasks in multi-obstacle environments.

[0077] 3. Encirclement time Encirclement time refers to the average time (in seconds) required for the pursuit vessel to successfully encircle the target. This metric reflects the efficiency of the algorithm in performing the encirclement task; the shorter the time, the higher the algorithm's execution efficiency. The evaluation was conducted by grouping 100 episodes into 10 independent tests, and the average of these tests was taken as the final result.

[0078] like Figure 13 As shown, with the increase in the number of obstacles, the encirclement time of other algorithms all show an upward trend, while the encirclement time of the RA-MADDPG algorithm also increases, but the increase is small and remains at a low level.

[0079] With 10 obstacles, the RA-MADDPG algorithm has a wraparound time of 33 seconds, which is about 17.4% less than the baseline MADDPG's 40.1 seconds, about 14.5% less than MASAC's 38.7 seconds, and about 15.4% less than MATD3's 39 seconds.

[0080] In summary, the RA-MADDPG algorithm demonstrates excellent encirclement time under different obstacle numbers, effectively reducing encirclement time and improving execution efficiency, showing its potential and advantages in pursuit tasks in multi-obstacle environments.

[0081] 4. Collision rate The collision rate is calculated as the ratio of the number of episodes where the mission failed due to collisions to the total number of evaluated episodes, reflecting the frequency of failures during mission execution. A low collision rate means that the pursuit vessel has higher safety and reliability in complex environments. Figure 14 As shown, the collision rate of all algorithms increases with the number of obstacles. However, RA-MADDPG consistently has a lower collision rate than other algorithms, especially when there are many obstacles, this advantage is more pronounced.

[0082] With 10 obstacles, the RA-MADDPG achieved a collision rate of 4%, a reduction of approximately 77.8% compared to MADDPG's 18%, approximately 66.7% compared to MASAC's 12%, and approximately 73.4% compared to MATD3's 15.4%. These data demonstrate that RA-MADDPG has a significant advantage in collision avoidance.

[0083] 5. Performance visualization analysis In a scenario containing 10 static obstacles, we conducted a comparative test on the obstacle avoidance, navigation, and collaborative control capabilities of each algorithm. Figure 15 The path trajectory shown demonstrates that the RA-MADDPG algorithm generates the most compact and smooth track, significantly reducing sharp turns and achieving precise obstacle avoidance and efficient target approach. Figure 16 As shown, although all algorithms eventually converge to near the desired bounding radius, RA-MADDPG achieves the minimum steady-state error. Similarly, Figure 17 The results show that agents controlled by RA-MADDPG can maintain smoother velocity curves and superior motion stability.

[0084] The quantitative analysis results in Table 2 further validate the above observations. The average path length (3.92) achieved by RA-MADDPG is significantly better than that of MADDPG (6.30) and MATD3 (4.43); its extremely low velocity standard deviation (0.0309) confirms its stable motion control capability, while the smallest encirclement radius error (0.0097) highlights its high precision advantage.

[0085] Table 2 Performance of the four algorithms

[0086] In summary, visual evidence and quantitative data together demonstrate that RA-MADDPG achieves superior adaptability and control accuracy in multi-agent navigation tasks.

[0087] III. Conclusion This invention addresses the challenge of balancing environmental constraints and collaborative efficiency in multi-unmanned surface vessel cooperative encirclement missions under dynamic obstacle environments. It proposes a multi-agent reinforcement learning framework that integrates a relationship-aware mechanism. A Relationship-Aware Attention Critics (RAAC) network is developed to optimize the cooperative strategy, a sparse proximity-aware observation model is constructed to achieve efficient state representation, and a stage-adaptive reward mechanism is designed to balance encirclement efficiency and obstacle avoidance safety. Simulation experiments in a three-chase-one-escape scenario validate the framework's effectiveness, demonstrating significant improvements in both success rate and collision avoidance.

[0088] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multi-unmanned surface vehicle cooperative hunting method for a multi-obstacle environment, characterized in that, The method comprises the following steps: A multi-unmanned surface vehicle cooperative hunting system model is established, the system model comprising a target hunting scene model, an unmanned surface vehicle kinematics model and a target escape strategy model; wherein the target hunting scene model defines the geometric and distance conditions for successful hunting, the unmanned surface vehicle kinematics model describes the position, heading angle, linear velocity and angular velocity changes of the unmanned surface vehicle through a differential equation, and the target escape strategy model calculates the repulsive force of the target from the hunting vehicles and obstacles based on the artificial potential field method to determine the target motion direction; An RA-MADDPG algorithm is designed, the RA-MADDPG algorithm comprising a relationship-aware attention critic network, a sparse neighbor-aware perception observation model and a stage adaptive reward mechanism; Based on the unmanned surface vehicle cooperative hunting system model and the RA-MADDPG algorithm, an experience replay pool is constructed and the algorithm is trained, the relationship-aware attention critic network parameters are updated by minimizing the temporal difference error, the Actor network parameters are updated by the policy gradient, and the target network parameters are updated by soft update; The trained RA-MADDPG algorithm is deployed to the multi-unmanned surface vehicles to control the multi-unmanned surface vehicles to perform a cooperative hunting task in a multi-obstacle environment, achieving rapid surrounding and stable maintenance of the target.

2. The method for multi-unmanned surface vehicle cooperative hunting in a multi-obstacle environment according to claim 1, wherein, The construction process of the relationship-aware attention critic network comprises: observed vector of all unmanned surface vehicles with corresponding action concatenating, obtaining an encoder input wherein , is a dimension of the observed vector, is a dimension of the action vector; will be projected to the hidden layer features by a fully connected layer where , is a weight matrix, is a bias term, is the hidden layer dimension; Setting an individual attention head, generating diversified feature representations wherein is a weight matrix of the th attention head, is a bias term of the th attention head; Concatenating vectors of actions of other unmanned surface vehicles than the current unmanned surface vehicle Encoding as decoder state wherein , is a decoder weight matrix, is a decoder bias term; Compute interaction score of each attention head with decoder state and normalize by Softmax function to get attention weights ; Context vectors are obtained by attention-weighted aggregation , the input two fully connected layer output value estimates.

3. The method for multi-UAVS cooperative hunting in a multi-obstacle environment according to claim 1, wherein, In the sparse proximity-aware observation model, the observation space of each unmanned surface vehicle is composed of its own motion state , teammate state , target state and adjacent obstacle state , that is ; wherein: The , , are position coordinates of the unmanned surface vehicle, is a scene normalization reference length, is a velocity direction angle, is a velocity module length, is a maximum speed of the unmanned surface vehicle; The include relative position, relative speed modulus and relative speed direction between unmanned surface vehicle and teammates. The include relative position, relative velocity modulus and relative velocity direction between the unmanned surface vehicle and the target. The wherein is the relative position of the unmanned surface vehicle to the closest obstacle, is the radius of the closest obstacle, and only when the closest obstacle is within the unmanned surface vehicle's perception radius is non-zero, otherwise set to 0.

4. The method for multi-UAVS cooperative hunting in a multi-obstacle environment according to claim 1, wherein, The stage adaptive reward mechanism is based on a "approach-surround-maintain" three-stage division, and the stage transition is determined according to the relative position relationship between the unmanned surface vehicle and the target: When there is at least one unmanned surface vehicle whose distance to the target > Enclosure phase threshold radius Enter the approach phase; When all unmanned surface vehicles are at a distance from the target ≤ the encircle phase threshold radius and the encircle condition is not met, enter the encircle phase, wherein > the expected capture radius for encircle success ; When all the unmanned surface vehicles simultaneously meet the hunting conditions, the maintain stage is entered; The trapping condition is and , is an unmanned surface vehicle and the relative bearing of the adjacent unmanned surface vehicle , , is the number of unmanned surface vehicles, is the angular tolerance error, is the distance tolerance error.

5. The method for multi-UAVS cooperative hunting in a multi-obstacle environment according to claim 1 or 4, characterized in that, The reward function in the stage-adaptive reward mechanism wherein is a global penalty term, is a stage-wise reward term; The ; The penalty for collision, when the distance of the unmanned surface vehicle from the obstacle ≤ obstacle safety distance , , when the distance between the unmanned surface vehicles ≤ safety distance , , where , is the penalty coefficient, is the prevention zero constant; The is a time penalty, wherein is a current time step, is a maximum time step, is a time penalty coefficient.

6. The method for multi-unmanned surface vehicle cooperative hunting in a multi-obstacle environment according to claim 5, wherein, The phased reward item The expression for the phase of the envelope is ; The wherein is a distance maintenance coefficient, and is 1 / 3 of the approach phase distance coefficient, denotes the distance of the unmanned surface vehicle from the target, is the desired capture radius; The wherein is the variance of the distances of all unmanned surface vehicles to the target, is the distance balancing coefficient, is the number of unmanned surface vehicles; The ,in Unmanned surface vessel Maximum angular deviation, The angular uniformity coefficient; The For alignment reward, when and , , otherwise 0, where is the distance tolerance, is the angle tolerance.

7. The method for multi-UAVS cooperative hunting in a multi-obstacle environment according to claim 6, wherein, The phased reward item The expression in the maintenance phase is ; The To fix the maintenance reward, ; The For a successful award, when the time that the unmanned surface vehicle maintains a pursuit state is greater than 0, otherwise 0.

8. The method for multi-unmanned surface vehicle cooperative hunting in a multi-obstacle environment according to claim 1, wherein, The differential equation of the unmanned surface vehicle kinematics model is: ； wherein, denotes the position coordinate of the unmanned surface vehicle in a two-dimensional plane; is the heading angle defined as the angle between the longitudinal axis of the vehicle and the positive direction of the x-axis of the global coordinate system; is the linear velocity along the heading direction; is the angular velocity; is the actual linear acceleration command; is the actual angular acceleration command, is the maximum linear acceleration, is the maximum angular acceleration.

9. The method for multi-unmanned surface vehicle cooperative hunting in a multi-obstacle environment according to claim 1, wherein, In the target escape strategy model, the total repulsive force suffered by the target ; wherein , , is the target position, is the unmanned surface vehicle position, is the unmanned surface vehicle distance to the target, is the obstacle position, is the obstacle distance to the target, is the number of unmanned surface vehicles, is the number of obstacles, , is the repulsive force coefficient; When the target instantaneous speed is greater than the maximum linear acceleration, the target maintains the original direction of motion, where is the maximum linear acceleration.

10. The method of claim 1, wherein, The process of constructing the experience replay pool and training the algorithm based on the unmanned surface vehicle cooperative hunting system model and the RA-MADDPG algorithm, updating the relationship-aware attention critic network parameters by minimizing the temporal difference error, updating the Actor network parameters by the policy gradient, and updating the target network parameters by soft update comprises: a relational-aware attention critic network by minimizing a loss function of a time-difference error updating parameters, wherein are relational-aware attention critic network parameters for a th unmanned surface vehicle; are expected operators over an experience replay pool, are local observation sets for all unmanned surface vehicles at a current time instant, are action sets for all unmanned surface vehicles at a current time instant, , is an immediate reward for a th unmanned surface vehicle at a current time instant, are local observation sets for all unmanned surface vehicles at a next time instant, is an experience replay pool; are target values for a th unmanned surface vehicle, , is an immediate reward for a th unmanned surface vehicle, is a discount factor, are target relational-aware attention critic network value estimates for a th unmanned surface vehicle, are target relational-aware attention critic network parameters, are action sets for all unmanned surface vehicles at a next time instant; are current relational-aware attention critic network value estimates for a th unmanned surface vehicle; Actor networks utilize policy gradients. Update parameters, where For the first The policy gradient of an unmanned surface vessel in the Actor network. For the "first" The gradient operator for calculating the partial derivatives of the Actor network parameters of an unmanned surface vessel. For the long-term expected return of the Actor network; For the observation expectation operator on the experience replay pool; For the first The gradient of the policy function of an unmanned surface vessel with respect to its own parameters. For the first Actor network policy function for an unmanned surface vessel For the first Local observations of an unmanned surface vessel; For the first The relationship between unmanned surface vessels, perception of attention, commentators, network The gradient of the value with respect to its own action. For the first The relationship between unmanned surface vessels, perception of attention, commentators, network Value functions; For the first The actions of an unmanned surface vessel; The target network parameters are updated by a soft update: wherein is a soft update coefficient, is a current relational-aware attention critic network parameter, is a target relational-aware attention critic network parameter, is a current Actor network parameter, is a target Actor network parameter.

Images