A multi-unmanned ship cooperative hunting method fusing graph neural network and attention
By integrating graph neural networks and attention into a multi-unmanned surface vessel (USV) collaborative encirclement method, the problem of low collaboration efficiency of multiple USVs in complex marine environments is solved, and efficient and stable multi-USV collaborative encirclement is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DALIAN MARITIME UNIVERSITY
- Filing Date
- 2026-05-15
- Publication Date
- 2026-06-26
AI Technical Summary
Existing methods for multi-UAV collaborative encirclement and capture lack flexibility in complex marine environments, making it difficult to efficiently utilize the interactive information between UAVs and adequately cope with dynamic changes in targets and multi-target scenarios, resulting in low collaborative efficiency and unstable encirclement formations.
A collaborative encirclement method for multiple unmanned surface vessels (USVs) is adopted, which integrates graph neural networks and attention. By constructing graph neural networks and multi-agent reinforcement learning models, the graph neural network is used to perform weighted feature aggregation, and a value network with a multi-head self-attention mechanism is introduced to adaptively focus on key interaction relationships in the encirclement process, thereby improving global situational awareness and collaborative decision-making capabilities.
It significantly enhances the global situational awareness and collaborative decision-making efficiency of multiple unmanned surface vessels in complex encirclement scenarios, improves the ability to identify key interaction relationships and the stability of collaborative encirclement, and enables efficient collaborative encirclement in real marine environments with island and reef obstacles and dynamic escape of targets.
Smart Images
Figure CN122284341A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent control technology for unmanned vessels, and in particular to a method for collaborative encirclement and capture of multiple unmanned vessels that integrates graph neural networks and attention. Background Technology
[0002] Unmanned surface vessels (USVs), as small, highly flexible, and fast-responding intelligent marine equipment, can perform various tasks such as exploration, defense, and rescue in complex environments such as the ocean. During encirclement missions, USVs need to perceive their surrounding environment and dynamic targets, and achieve target encirclement through cooperative strategies. However, the complex and ever-changing marine environment, with its islands, reefs, and unpredictable dynamic escape targets, presents numerous challenges for USVs in cooperative encirclement missions. Currently, traditional methods for multi-USV cooperative encirclement missions mainly rely on geometric algorithms, bio-inspired strategies, or force field-based control methods. These methods typically rely on precise environmental modeling or pre-defined rules to achieve encirclement of a single target or in specific scenarios. However, these methods lack flexibility in complex marine environments, struggle to efficiently utilize the interactive information between USVs, and are unable to adequately address dynamic target changes and multi-target scenarios, resulting in low cooperation efficiency and unstable encirclement formations.
[0003] In recent years, deep reinforcement learning methods have emerged as an effective approach to solving the problem of cooperative encirclement of multiple USVs, thanks to their ability to autonomously learn strategies from environmental interaction data. Deep reinforcement learning-based methods can extract key features in high-dimensional state spaces, enabling efficient collaboration among multiple agents. However, existing learning methods still have shortcomings in global state perception, modeling of relationships between agents, and attention to key interaction information, resulting in unstable encirclement strategies, limited generalization performance, and difficulty in meeting the complex collaborative needs of real-world marine environments. Summary of the Invention
[0004] This invention provides a collaborative encirclement and capture method for multiple unmanned surface vessels that integrates graph neural networks and attention, in order to overcome the above-mentioned technical problems.
[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A collaborative encirclement and capture method for multiple unmanned surface vessels integrating graph neural networks and attention, comprising the following steps: S1. Construct a multi-unmanned surface vessel (USV) collaborative encirclement and capture mission scenario involving multiple USVs and an escape target, and initialize the pursuit and escape states of the USVs and the escape target; establish the kinematic model of the USVs and the motion strategy of the escape target. S2. Obtain the global state based on the multi-unmanned surface vessel collaborative encirclement mission scenario and the current pursuit status, obtain the observation information of each pursuit unmanned surface vessel based on the global state, and calculate the encirclement characteristics of the pursuit unmanned surface vessel based on the observation information. S3. Establish a graph neural network based on the encirclement features of the pursuit unmanned vessel, perform feature weighting and aggregation based on the graph neural network, and output the graph aggregated features; S4. Establish a multi-agent reinforcement learning model and input the graph aggregation features into the multi-agent reinforcement learning model; The multi-agent reinforcement learning model includes a policy network and a value network based on a multi-head self-attention mechanism. The policy network is used to output the control actions for pursuing the unmanned surface vessel, and the value network is used to output the state value estimate. S5. Combine the control actions of the pursuit unmanned surface vessel with the kinematic model of the pursuit unmanned surface vessel and apply them to the environment. At the same time, update the state of the escape target according to the escape target's motion strategy to obtain a new global state. Calculate the reward value based on the encirclement situation reflected by the new global state, and store the current global state, control actions, action probabilities, reward value, and state value estimate into the experience replay pool. S6. Determine whether the samples in the experience replay pool have reached the preset size. If not, use the new global state as the current pursuit state and return to S2 to continue execution. If they have reached the preset size, update the policy network and value network parameters using the samples in the experience replay pool. S7. Determine whether the preset termination condition has been met. If not, return to S2 to continue the loop; if it has been met, terminate the training, and finally output the trained policy network. Use the control actions output by the policy network to achieve multi-unmanned surface vessel cooperative encirclement and capture.
[0006] Furthermore, the established kinematic model for pursuing unmanned surface vessels and the escape target movement strategy include: The kinematic model of the pursuit unmanned surface vessel is represented as follows: (1) and ; In the formula, , Indicates the first The location coordinates of the pursuit drone. Indicates speed, Indicates the heading angle. Indicates angular velocity, This represents the first derivative of the variable with respect to time. Represents linear acceleration. Indicates angular acceleration; This represents the maximum speed. This represents the maximum angular velocity. The escape target is controlled using heuristic escape maneuver strategies, including: The encirclement center was calculated based on the positions of all pursuing unmanned vessels at the two most recent moments: (2) (3) In the formula, and They represent the first A pursuit drone at all times and Location, and These represent the times of all the pursuit drones. and The average position Indicates the center of the encirclement; Indicates the number of unmanned vessels being pursued; and Indicates weight; Set the escape target at time The position is Then, the unit direction away from the center of the encirclement is represented as: (4) Based on this, a random angle perturbation is introduced: (5) In the formula, Indicates at time A random angle; Based on the unit direction away from the center of the encirclement and random angle perturbation The final escape direction of the escape target is: (6).
[0007] Furthermore, in S2, the specific steps for obtaining the global state based on the multi-unmanned surface vessel (USV) collaborative encirclement mission scenario and the current pursuit status, obtaining the observation information of each pursuit USV based on the global state, and calculating the encirclement characteristics of the pursuit USV based on the observation information include: The coordinated encirclement task is modeled as a Markov decision process, and a state space is defined. This represents the global state at each moment; Set the observation model as The global state is mapped to local observations of each unmanned surface vessel (USV) using an observation model, thereby outputting the observation set of each pursuit USV. , This indicates the observation information of each pursuit drone; Based on the observation information of the pursuit unmanned surface vessel, the basic and extended features are calculated, including: The fundamental features include distance balance. and relative motion trend Among them, distance uniformity is used to measure the consistency of the distance distribution from each pursuing unmanned surface vessel to the escaping target, and the calculation formula is: (7) In the formula, Indicates the first The distance between the pursuing unmanned vessel and the escaping target. Indicates the number of unmanned vessels being pursued; The relative motion trend is used to characterize the degree of consistency between the current direction of motion of the escaping target and the direction of the encirclement center. The calculation formula is as follows: (8) In the formula, Indicates the geometric center of the pursuit unmanned surface vessel formation. , Indicates the first i The location of the pursuing unmanned vessel Indicates the direction of movement of the escaping target. , Indicates the heading angle of the escape target. Indicates the location of the escape target; The extended features include closure, maximum capture gap, capture quality, symmetry, distribution balance, and coverage integrity, among which, the first... k The formula for calculating the closure degree of a single pursuit unmanned surface vessel is: (9) In the formula, Indicates the first j The location of the pursuing unmanned vessel To pursue the number of unmanned boats; The maximum gap in the encirclement is used to extract the weakest link in the coordination during the encirclement. k The formula for calculating the maximum gap in the encirclement of a single pursuit unmanned vessel is: (10) The encirclement quality is used to characterize the overall encirclement quality when the escaping target deviates from the encirclement area. k The formula for calculating the capture mass of a single pursuit unmanned surface vessel is: (11) The symmetry is used to measure the relative centrality of the pursuit unmanned surface vessel within the formation. k The formula for calculating the symmetry of a single pursuit unmanned surface vessel is: (12) in, Let represent the set of numbered pairs consisting of all the other pursuit unmanned vessels except for the k-th one. Represents a set The number of numbered pairs Indicates the position of the k-th pursuit drone; The distribution uniformity is used to measure the uniformity of the angular distribution of each pursuit drone around the escaping target. k The formula for calculating the distribution balance of the pursuit unmanned surface vessels is: (13) In the formula, This indicates the angular interval between each pursuing unmanned vessel and the escaping target; The coverage integrity is used to characterize the largest uncovered corner area of the encirclement formation. k The formula for calculating the coverage integrity of a single pursuit drone is: (14) Therefore, the first The characteristics of the pursuit of unmanned surface vessels are represented as follows: (15) in, This represents the set of basic motion state features of the k-th pursuit unmanned surface vessel. Let K represent the set of extended encirclement geometric features of the k-th pursuit unmanned vessel. Including distance balance and relative motion trend , Including closure The largest gap in the encirclement Encirclement quality Symmetry Distribution balance and coverage completeness : (16).
[0008] Furthermore, in S3, the specific steps of establishing a graph neural network based on the encirclement features of the pursuit unmanned surface vessel, performing feature weighting aggregation based on the graph neural network, and outputting the graph aggregation features include: Building a graph neural network: At each decision-making moment, the pursuing unmanned surface vessel and the escaping target are abstracted as nodes in a graph neural network, and a directed graph is constructed, represented as: (17) In the formula, Indicates the escape target node; Represents a set of nodes. Represents the set of edges; The relative relationships between the nodes pursuing the unmanned surface vessel and the escape target node are used as graph edges, i.e., the adjacency matrix is defined as follows: (18) In the formula, This is the adjacency distance threshold; Based on the graph neural network, feature weighting aggregation is performed, and the output graph aggregated features include: By using a pre-built multilayer perceptron, the features of the pursuing unmanned surface vessel node and the escape target node in the graph neural network are nonlinearly encoded to obtain the encoded node hidden representations: (19) In the formula, Represents the ReLU activation function; and These represent the weight matrix and bias of the first fully connected layer in a multilayer perceptron, respectively. Represents a node v The original features have a dimension of 8; and These represent the weight matrix and bias of the second fully connected layer in a multilayer perceptron, respectively. Represents a 64-dimensional vector space; By using a pre-defined two-layer graph attention layer to perform weighted aggregation of adjacent node information, the updated node hidden representation is obtained. ; The updated node representation is hidden through a pre-defined output layer. The compressed graph aggregation features, represented as: (20) In the formula, and These represent the weight matrix and bias of the first fully connected layer in the output layer, respectively. and These represent the weight matrix and bias of the second fully connected layer in the output layer, respectively. This represents a 32-dimensional vector space.
[0009] Furthermore, in S4, the control actions of the pursuit unmanned surface vessel output by the policy network are represented as follows: (twenty one) In the formula, Indicates speed control quantity. Indicates the angular velocity control quantity; The specific steps for estimating the output state value of the value network include: The query vector, key vector, and value vector are calculated and represented as follows: (twenty two) In the formula, Input features, i.e., graph aggregation features ; These represent the weight matrices for the query vector, key vector, and value vector, respectively. The attention weights are calculated as follows: (twenty three) In the formula, The feature dimensions represent the query vector and key vector; Represents the transpose of the key vector; Based on the above attention weights, the value vector V Perform weighted aggregation to obtain the output of each head; The outputs of each head are concatenated and linearly transformed to obtain the attention interaction features of all the pursuit unmanned surface vessels. The attention interaction features are subjected to residual connection and layer normalization to obtain the first fusion feature; The first fused feature is fed into a feedforward network consisting of two fully connected layers, ReLU activation and Dropout to perform nonlinear transformation and obtain nonlinear transformation features. The nonlinear transformation feature and the first fusion feature are subjected to residual connection and layer normalization to obtain the second fusion feature; The second fused feature is flattened and concatenated along the feature dimension, and then input into the value output network consisting of multiple fully connected layers and Tanh activation function to finally obtain the state value estimate.
[0010] Furthermore, in S5, the reward value is calculated based on the encirclement situation reflected by the new global state, including: At each moment, the reward value for each pursuit drone is calculated based on a set reward function, which is expressed as: (twenty four) Among them, the reward for encirclement and capture Balanced rewards based on distance Evenly distributed angles reward Complete capture reward Step length penalty and collision penalty Weighted composition: (25) (26) (27) (28) In the formula, These are the weighting coefficients; This represents the average distance from all pursuing unmanned vessels to the escaping target. This represents the standard deviation of the corresponding distance. Indicates the radius of the encirclement. This represents the angular interval between adjacent pursuit unmanned surface vessels after sorting by azimuth; step size penalty. Fixed negative reward, collision penalty Take the penalty value when a collision occurs; Successful capture reward Represented as: (29) In the formula, This indicates the number of consecutive steps that meet the capture conditions. This indicates the number of consecutive steps required to determine if the encirclement is successful. Indicates the maximum number of steps. Indicates the current moment.
[0011] Beneficial Effects: This invention establishes a graph neural network based on the encirclement features of unmanned surface vessels (USVs), and performs weighted feature aggregation based on the graph neural network to output graph aggregated features. This solves the problem of input features being sensitive to changes in the number and order of USVs in traditional methods, significantly improving the global situational awareness of multiple USVs in complex encirclement scenarios and providing structured feature support for collaborative decision-making. The introduction of a value network based on a multi-head self-attention mechanism into the multi-agent reinforcement learning model can adaptively focus on key interactions during the encirclement process and accurately assess the state value of the current encirclement situation. This invention is designed for real marine environments with island and reef obstacles and dynamic target escape, not only improving the efficiency of collaborative decision-making but also enhancing the global situational awareness, key interaction identification, and stability of collaborative encirclement among multiple USVs. Attached Figure Description To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a first flowchart of a multi-unmanned surface vessel cooperative encirclement method that integrates graph neural networks and attention in this invention; Figure 2 This is a general framework diagram of the multi-unmanned surface vessel cooperative encirclement method in an embodiment of the present invention; Figure 3 This is a schematic diagram of the second process of the multi-unmanned surface vessel cooperative encirclement method in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the principle of multi-unmanned surface vessel collaborative encirclement and capture in an embodiment of the present invention; Figure 5 This is a schematic diagram of a pursuit scenario between an unmanned surface vessel and an escaping target in an embodiment of the present invention; Figure 6 The above-ground map of the actual sea area selected in this embodiment of the invention; Figure 7 This is a schematic diagram of the coordinated pursuit trajectory of three pursuit unmanned vessels to capture one escaped target in a real marine environment, as described in an embodiment of the present invention. Figure 8 This is a schematic diagram of the coordinated pursuit trajectory of 5 pursuit unmanned surface vessels to capture 1 escaped target in a real marine environment, as described in an embodiment of the present invention. Figure 9 This is a schematic diagram of the coordinated pursuit trajectory of 8 pursuit unmanned surface vessels to capture 2 escaped targets in a real marine environment, as described in an embodiment of the present invention. Detailed Implementation
[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0014] This embodiment provides a collaborative encirclement method for multiple unmanned surface vessels (USVs) that integrates graph neural networks and attention, such as... Figures 1 to 3 As shown, the specific steps include: S1. Construct a multi-unmanned surface vessel (USV) collaborative encirclement and capture mission scenario involving multiple USVs and an escape target, and initialize the pursuit and escape states of the USVs and the escape target; establish the kinematic model of the USVs and the motion strategy of the escape target. In a specific embodiment, S1, constructing a multi-unmanned surface vessel (USV) collaborative encirclement and capture mission scenario involving multiple USVs pursuing and capturing an escaping target includes: In a multi-unmanned surface vessel (USV) collaborative encirclement and capture mission scenario, N pursuit USVs and one or more escape targets are set up. The pursuit USVs and escape targets have opposite mission objectives. The pursuit USVs need to complete the encirclement and capture of the escape targets as quickly as possible while avoiding obstacles in the sea area, while the escape targets should move as far away from the pursuit USVs as possible. In this embodiment, the multi-USV collaborative encirclement and capture mission scenario is constructed based on real nautical charts or real sea area satellite images, so that the scenario includes the actual outlines of islands and reefs, the boundaries of navigable waters, and target waypoint information. like Figure 4As shown, during the coordinated encirclement process, the Apollonius circle serves as the geometric constraint basis for the encirclement. For multiple pursuing unmanned surface vessels (USVs) and the same escaping target, the encirclement boundary can be dynamically constructed based on the geometric relationship between each USV and the target. When multiple USVs are tangent and form a closed region containing the escaping target, an effective encirclement can be achieved, thereby compressing the target's maneuvering space and restricting its escape direction.
[0015] In this embodiment, the established kinematic model for pursuing unmanned surface vessels and the escape target motion strategy include: The kinematic model of the pursuit unmanned surface vessel is represented as follows: (1) In the formula, , Indicates the first The location coordinates of the pursuit drone. Indicates speed, Indicates the heading angle. Indicates angular velocity, This represents the first derivative of the variable with respect to time. Represents linear acceleration. Indicates angular acceleration; Furthermore, constraints are set on the velocity and angular velocity of the kinematic model of the pursuit unmanned surface vessel to ensure that they meet the following conditions. and , This represents the maximum speed. This is the maximum angular velocity, ensuring that the encirclement and control results meet the motion constraints of the unmanned surface vessel. The escape target is controlled using heuristic escape maneuver strategies, including: The encirclement center was calculated based on the positions of all pursuing unmanned vessels at the two most recent moments: (2) (3) In the formula, and They represent the first A pursuit drone at all times and Location, and These represent the times of all the pursuit drones. and The average position Indicates the center of the encirclement; Indicates the number of unmanned vessels being pursued; and Indicates weight; Set the escape target at time The position is Then, the unit direction away from the center of the encirclement is represented as: (4) Based on this, a random angle perturbation is introduced: (5) In the formula, Indicates at time A random angle; Based on the unit direction away from the center of the encirclement and random angle perturbation The final escape direction of the escape target is: (6).
[0016] Preferably, in addition to the basic direction away from the center of the encirclement, this embodiment also sets a preset waypoint for the escape target, so that it moves towards the next waypoint while moving away from the center of the encirclement, thereby forming an escape trajectory that is both planned and random in complex sea areas.
[0017] S2. Obtain the global state based on the multi-unmanned surface vessel collaborative encirclement mission scenario and the current pursuit status, obtain the observation information of each pursuit unmanned surface vessel based on the global state, and calculate the encirclement characteristics of the pursuit unmanned surface vessel based on the observation information. In a specific embodiment, S2, the steps of obtaining a global state based on the multi-unmanned surface vessel (USV) collaborative encirclement mission scenario and the current pursuit status, obtaining observation information of each pursuit USV based on the global state, and calculating the encirclement characteristics of the pursuit USV based on the observation information include: The coordinated encirclement task is modeled as a Markov decision process, and a state space is defined. This represents the global state at each moment; Set the observation model as The global state is mapped to local observations of each unmanned surface vessel (USV) using an observation model, thereby outputting the observation set of each pursuit USV. , This indicates the observation information of each pursuit drone; Specifically, the state characteristics of each pursuit unmanned vessel include two parts: basic characteristics and extended characteristics. The basic characteristics are used to characterize the encirclement distance relationship and relative motion trend, while the extended characteristics are used to characterize the degree of encirclement closure, encirclement gap, encirclement quality, and spatial distribution balance.
[0018] In this embodiment, as Figure 5 In the pursuit scenario shown, U1, U2, and U3 represent three pursuit unmanned surface vessels (USVs), and T represents the escaping target. Based on the observation information of the USVs, basic and extended features are calculated, including: The fundamental features include distance balance. and relative motion trend Among them, distance uniformity is used to measure the consistency of the distance distribution from each pursuing unmanned surface vessel to the escaping target, and the calculation formula is: (7) In the formula, such as Figure 5 As shown Indicates the first The distance between the pursuing unmanned vessel and the escaping target. Indicates the number of unmanned boats being pursued. The smaller the value, the more even the distances between each pursuing unmanned surface vessel and the target. The relative motion trend is used to characterize the degree of consistency between the current direction of motion of the escaping target and the direction of the encirclement center. The calculation formula is as follows: (8) In the formula, Indicates the geometric center of the pursuit unmanned surface vessel formation. , Indicates the first i The location of the pursuing unmanned vessel Indicates the direction of movement of the escaping target. ,like Figure 5 As shown, Indicates the heading angle of the escape target. It represents the location of the escaping target; by introducing the relative motion trend, the impact of the target's escape behavior on the capture strategy can be reflected in the state space. The extended features include closure, maximum capture gap, capture quality, symmetry, distribution balance, and coverage integrity, among which, the first... k The formula for calculating the closure degree of a single pursuit unmanned surface vessel is: (9) In the formula, Indicates the first j The location of the pursuing unmanned vessel To pursue the number of unmanned boats; Specifically, the closure degree is used to characterize the projection deviation of the escape target relative to the midpoint of the line connecting each pursuing unmanned surface vessel, thereby reflecting the degree of constraint of the current encirclement formation on the escape target.
[0019] The maximum gap in the encirclement is used to extract the weakest link in the coordination during the encirclement. k The formula for calculating the maximum gap in the encirclement of a single pursuit unmanned vessel is: (10) The encirclement quality is used to characterize the overall encirclement quality when the escaping target deviates from the encirclement area.k The formula for calculating the capture mass of a single pursuit unmanned surface vessel is: (11) Specifically, by introducing the maximum catch gap and catch quality, strategies can be guided to prioritize filling weak areas in the catch and improve the overall catch effect.
[0020] The symmetry is used to measure the relative centrality of the pursuit unmanned surface vessel within the formation. k The formula for calculating the symmetry of a single pursuit unmanned surface vessel is: (12) in, Let represent the set of numbered pairs consisting of all the other pursuit unmanned vessels except for the k-th one. Represents a set The number of numbered pairs Indicates the position of the k-th pursuit drone; The distribution uniformity is used to measure the uniformity of the angular distribution of each pursuit drone around the escaping target. k The formula for calculating the distribution balance of the pursuit unmanned surface vessels is: (13) In the formula, such as Figure 5 As shown, This indicates the angular interval between each pursuing unmanned vessel and the escaping target; The coverage integrity is used to characterize the largest uncovered corner area of the encirclement formation. k The formula for calculating the coverage integrity of a single pursuit drone is: (14) Specifically, symmetry, distribution balance, and coverage integrity can further ensure the uniformity and closed-loop nature of the encirclement formation.
[0021] Therefore, the first The characteristics of the pursuit of unmanned surface vessels are represented as follows: (15) in, This represents the set of basic motion state features of the k-th pursuit unmanned surface vessel. Let K represent the set of extended encirclement geometric features of the k-th pursuit unmanned vessel. Including distance balance and relative motion trend , Including closure The largest gap in the encirclement Encirclement quality Symmetry Distribution balance and coverage completeness : (16) Specifically, through the above-mentioned encirclement feature design, the geometric relationship of the Apollonius circle encirclement, the target escape trend, and the multi-vehicle cooperative distribution relationship can be jointly encoded into the input of the strategy network. S3. Establish a graph neural network based on the encirclement features of the pursuit unmanned vessel, perform feature weighting and aggregation based on the graph neural network, and output the graph aggregated features; In a specific embodiment, S3, the specific steps of establishing a graph neural network based on the encirclement features of the pursuit unmanned surface vessel, performing feature weighting aggregation based on the graph neural network, and outputting the graph aggregated features include: Building a graph neural network: like Figure 3 As shown, at each decision-making moment, the pursuing unmanned surface vessel and the escaping target are abstracted as nodes in a graph neural network, and a directed graph is constructed, represented as follows: (17) In the formula, Indicates the escape target node; Represents a set of nodes. Represents the set of edges; The aforementioned encirclement features are used as the features of the unmanned surface vessel nodes being pursued. The features of the escape target nodes include the location, heading, speed, and node type of the escape target. The relative relationships between the nodes pursuing the unmanned surface vessel and the escape target node are used as graph edges, i.e., the adjacency matrix is defined as follows: (18) In the formula, This is the adjacency distance threshold; Through the above graph neural network modeling, the spatial relationships between multiple pursuit unmanned surface vessels and between pursuit unmanned surface vessels and escape targets can be explicitly represented; Based on the graph neural network, feature weighting aggregation is performed, and the output graph aggregated features include: By using a pre-built multilayer perceptron, the features of the pursuing unmanned surface vessel node and the escape target node in the graph neural network are nonlinearly encoded to obtain the encoded node hidden representations: (19) In the formula, Represents the ReLU activation function; and These represent the weight matrix and bias of the first fully connected layer in a multilayer perceptron, respectively. Represents a node v The original features have a dimension of 8; and These represent the weight matrix and bias of the second fully connected layer in a multilayer perceptron, respectively. Represents a 64-dimensional vector space; By using a pre-defined two-layer graph attention layer to perform weighted aggregation of adjacent node information, the updated node hidden representation is obtained. ; Specifically, through a pre-set two-layer graph attention layer, during the process of weighted aggregation of information of neighboring nodes based on adjacency relationship, the hidden representation of the escape target node will be used as adjacency information and fused into the hidden representation of the pursuit unmanned boat node through attention weights, so that the features of each pursuit unmanned boat node simultaneously encode its own state, teammate distribution and target position. The updated node representation is hidden through a pre-defined output layer. The compressed graph aggregation features, represented as: (20) In the formula, and These represent the weight matrix and bias of the first fully connected layer in the output layer, respectively. and These represent the weight matrix and bias of the second fully connected layer in the output layer, respectively. This represents a 32-dimensional vector space.
[0022] Specifically, by employing graph neural networks, the features of the subsequent input multi-agent reinforcement learning model can be made insensitive to changes in the number of unmanned surface vessels and the order of input, thereby improving the global state perception and decision-making coordination capabilities during multi-vessel coordinated encirclement and capture.
[0023] S4. Establish a multi-agent reinforcement learning model and input the graph aggregation features into the multi-agent reinforcement learning model; the multi-agent reinforcement learning model includes a policy network and a value network based on a multi-head self-attention mechanism, the policy network is used to output the control actions of the pursuit unmanned surface vessel, and the value network is used to output the state value estimate. In a specific embodiment, in S4, the control action of the pursuit unmanned surface vessel output by the policy network is represented as follows: (twenty one) In the formula, Indicates speed control quantity. Indicates the angular velocity control quantity; Specifically, the action space uses two-dimensional action vectors to represent the first... The control output of the pursuit unmanned surface vessel (USV) has an action space that reduces the difficulty of solving reinforcement learning problems under complex dynamic conditions, while simultaneously satisfying the control constraints of the USV. (See [link to relevant documentation]). Figure 3 The action module in [the context].
[0024] This embodiment introduces a multi-head self-attention mechanism into the value network; see [link / reference]. Figure 3 The multi-head self-attention mechanism module in the value network, wherein the value network focuses on graph aggregation features extracted by the graph neural network. The specific steps involved in processing the data to output a state value estimate include: The query vector, key vector, and value vector are calculated and represented as follows: (twenty two) In the formula, Input features, i.e., graph aggregation features ; These represent the weight matrices for the query vector, key vector, and value vector, respectively. The attention weights are calculated as follows: (twenty three) In the formula, The feature dimensions represent the query vector and key vector; Represents the transpose of the key vector; Based on the above attention weights, the value vector V We perform weighted aggregation to obtain the output of each head; we then concatenate the outputs of each head and perform a linear transformation to obtain the attention interaction features of all the pursuit drones. The attention interaction features are subjected to residual connection and layer normalization to obtain the first fusion feature; The first fused feature is fed into a feedforward network consisting of two fully connected layers, ReLU activation and Dropout to perform nonlinear transformation and obtain nonlinear transformation features. The nonlinear transformation feature and the first fusion feature are subjected to residual connection and layer normalization to obtain the second fusion feature, which is the high-order representation of each unmanned surface vessel. The second fused feature is flattened and concatenated along the feature dimension, and then input into the value output network consisting of multiple fully connected layers and Tanh activation function to finally obtain the state value estimate.
[0025] Specifically, by introducing a multi-head self-attention mechanism on the value network side, the multi-agent reinforcement learning model can adaptively focus on key pursuit unmanned vessels and their interaction relationships during the policy evaluation process, thereby improving the accuracy of value evaluation and the stability of policy updates.
[0026] S5. Combine the control actions of the pursuit unmanned surface vessel with the kinematic model of the pursuit unmanned surface vessel and apply them to the environment. At the same time, update the state of the escape target according to the escape target's motion strategy to obtain a new global state. Calculate the reward value based on the encirclement situation reflected by the new global state, and store the current global state, control actions, reward value, and state value estimate into the experience replay pool. In a specific embodiment, S5, calculating the reward value based on the encirclement situation reflected by the new global state includes: At each moment, the reward value for each pursuit drone is calculated based on a set reward function, which is expressed as: (twenty four) Among them, the reward for encirclement and capture Balanced rewards based on distance Evenly distributed angles reward Complete capture reward Step length penalty and collision penalty Weighted composition: (25) (26) (27) (28) In the formula, These are the weighting coefficients; This represents the average distance from all pursuing unmanned vessels to the escaping target. This represents the standard deviation of the corresponding distance. Indicates the radius of the encirclement. This represents the angular interval between adjacent pursuit unmanned surface vessels after sorting by azimuth; step size penalty. Fixed negative reward, collision penalty When a collision occurs, a penalty value is taken.
[0027] The above-mentioned reward design can guide multiple pursuit unmanned boats to gradually form a pursuit formation with balanced distances, uniform angle distribution, and a complete closed loop.
[0028] To encourage early completion of the encirclement and maintain continuous encirclement, a reward system for successful encirclements will be introduced. : (29) In the formula, This indicates the number of consecutive steps that meet the capture conditions. This indicates the number of consecutive steps required to determine if the encirclement is successful. Indicates the maximum number of steps. Indicates the current moment.
[0029] Specifically, the successful capture reward can enhance the strategy's preference for rapid and stable captures.
[0030] S6. Determine whether the samples in the experience replay pool have reached the preset size. If not, use the new global state as the current pursuit state and return to S2 to continue execution. If they have reached the preset size, update the policy network and value network parameters using the samples in the experience replay pool. Specifically, in this embodiment, the method proposed adopts a training framework of centralized training and distributed execution. For example... Figure 3 As shown, during the training phase, the environment outputs the global state. The observation model and reward function generate the observation sets for each pursuit drone. and corresponding rewards The graph neural network constructs a graph and encodes graph attention on the observation set to obtain the feature representations of each pursuit drone. The policy network is based on Output Action The environment is then updated to the next state based on the actions. The system caches the state, actions, rewards, and intermediate network data in an experience replay pool. Once a set scale is reached, the parameters of the policy network and value network are updated until a policy network that meets the requirements for the encirclement effect is obtained. During the execution phase, each pursuit drone achieves decentralized control based solely on local observations and the actions output by the policy network.
[0031] S7. Determine whether the preset termination condition has been met. If not, return to S2 to continue the loop. If it has been met, terminate the training and finally output the trained policy network. Use the control actions output by the policy network to achieve multi-unmanned surface vessel coordinated encirclement and capture.
[0032] Specifically, in multi-UAV cooperative encirclement missions, complex spatial relationships exist between UAVs and between UAVs and escaping targets. Directly concatenating the observation information from each UAV and inputting it into a multi-agent reinforcement learning model (MAPPO) would lead to a significant increase in input dimensionality with the number of UAVs, and make the network highly sensitive to input order, making it difficult to accurately describe the cooperative relationships between UAVs. This embodiment introduces a graph neural network, obtaining the final graph aggregation feature through node encoding, edge encoding, and graph attention aggregation. This enables the MAPPO policy network to output more reasonable cooperative actions based on the cooperative relationships between UAVs, improving its adaptability to different numbers and arrangements of UAVs. Furthermore, this embodiment introduces a multi-head self-attention mechanism into the MAPPO value network, allowing it to adaptively focus on UAVs with greater impact on the current encirclement effect, key relative motion relationships, and key geometric encirclement features, thereby improving global situational awareness and the ability to represent multi-agent cooperative relationships.
[0033] In this embodiment, as Figure 6 As shown, a real sea area was selected as the experimental sea area for modeling to verify the feasibility of the method proposed in this embodiment in complex marine environments. The experimental sea area includes multiple island and reef obstacles; preferably, the experimental sea area extends to the northern latitude... To the North Latitude East longitude East Longitude The constructed environment measures 3367m × 2148m. The experimental platform uses PyTorch for algorithm training, with a maximum training epoch of 10000, a maximum iteration step per epoch of 400, a maximum speed of 7m / s for the pursuit drone, a maximum angular velocity of 0.8rad / s, a discount factor of 0.99, a pruning factor of 0.2, an Actor network learning rate of 0.0003, a Critic network learning rate of 0.001, a capture radius of 100, and a batch size of 256.
[0034] In the first embodiment, three pursuit drones are deployed to surround and capture one escape target, such as... Figure 7 As shown, as the encirclement process progresses, the three pursuit drones gradually close in on the escape target, forming a relatively even encirclement distribution. Upon successful encirclement, the distances between each pursuit drone and the escape target tend to be consistent, indicating that the method proposed in this embodiment can achieve effective coordinated encirclement in a real marine environment.
[0035] In the second embodiment, five unmanned pursuit vessels are deployed to surround and capture one escaped target, such as... Figure 8As shown, compared to the first embodiment, with the increase in the number of pursuit unmanned surface vessels, the encirclement configuration is more complete and the degree of encirclement closure is higher. Each pursuit unmanned surface vessel can still automatically adjust its position relationship according to the learned strategy and gradually form a cooperative encirclement, indicating that the method proposed in this embodiment has good scalability.
[0036] In the third embodiment, eight unmanned pursuit vessels are deployed to surround and capture two escaped targets, such as... Figure 9 As shown, based on the target distribution and relative distance relationship, the eight pursuit unmanned surface vessels can adaptively allocate the targets to be pursued, forming two relatively independent collaborative pursuit subgroups, and each subgroup can carry out pursuit of the corresponding escape targets. Each subgroup can maintain a relatively stable pursuit configuration during the pursuit process, and finally achieve effective pursuit of the two escape targets. This shows that the method proposed in this embodiment also has good effectiveness and scalability in multi-target collaborative pursuit scenarios.
[0037] In summary, the multi-unmanned surface vessel (USV) collaborative encirclement method based on the Apollonius circle provided in this embodiment, by constructing encirclement geometric constraint features, modeling based on graph neural networks, and using a multi-head self-attention value evaluation mechanism, can achieve efficient and stable collaborative encirclement of multiple USVs in complex marine environments containing real island and reef obstacles and dynamically escaping targets.
[0038] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for collaborative encirclement and capture of multiple unmanned surface vessels by integrating graph neural networks and attention, characterized in that, The specific steps include: S1. Construct a multi-unmanned surface vessel (USV) collaborative encirclement and capture mission scenario involving multiple USVs and an escape target, and initialize the pursuit and escape states of the USVs and the escape target; establish the kinematic model of the USVs and the motion strategy of the escape target. S2. Obtain the global state based on the multi-unmanned surface vessel collaborative encirclement mission scenario and the current pursuit status, obtain the observation information of each pursuit unmanned surface vessel based on the global state, and calculate the encirclement characteristics of the pursuit unmanned surface vessel based on the observation information. S3. Establish a graph neural network based on the encirclement features of the pursuit unmanned vessel, perform feature weighting and aggregation based on the graph neural network, and output the graph aggregated features; S4. Establish a multi-agent reinforcement learning model and input the graph aggregation features into the multi-agent reinforcement learning model; The multi-agent reinforcement learning model includes a policy network and a value network based on a multi-head self-attention mechanism. The policy network is used to output the control actions for pursuing the unmanned surface vessel, and the value network is used to output the state value estimate. S5. Combine the control actions of the pursuit unmanned surface vessel with the kinematic model of the pursuit unmanned surface vessel and apply them to the environment. At the same time, update the state of the escape target according to the escape target's motion strategy to obtain a new global state. Calculate the reward value based on the encirclement situation reflected by the new global state, and store the current global state, control actions, action probabilities, reward value, and state value estimate into the experience replay pool. S6. Determine whether the samples in the experience replay pool have reached the preset size. If not, use the new global state as the current pursuit state and return to S2 to continue execution. If they have reached the preset size, update the policy network and value network parameters using the samples in the experience replay pool. S7. Determine whether the preset termination condition has been met. If not, return to S2 to continue the loop; if it has been met, terminate the training, and finally output the trained policy network. Use the control actions output by the policy network to achieve multi-unmanned surface vessel cooperative encirclement and capture.
2. The multi-unmanned surface vessel cooperative encirclement method integrating graph neural networks and attention as described in claim 1, characterized in that, The established kinematic model for pursuing unmanned surface vessels and the escape target movement strategies include: The kinematic model of the pursuit unmanned surface vessel is represented as follows: (1) and ; In the formula, , Indicates the first The location coordinates of the pursuit drone. Indicates speed, Indicates the heading angle. Indicates angular velocity, This represents the first derivative of the variable with respect to time. Represents linear acceleration. Indicates angular acceleration; This represents the maximum speed. This represents the maximum angular velocity. The escape target is controlled using heuristic escape maneuver strategies, including: The encirclement center was calculated based on the positions of all pursuing unmanned vessels at the two most recent moments: (2) (3) In the formula, and They represent the first A pursuit drone at all times and Location, and These represent the times of all the pursuit drones. and The average position Indicates the center of the encirclement; Indicates the number of unmanned vessels being pursued; and Indicates weight; Set the escape target at time The position is Then, the unit direction away from the center of the encirclement is represented as: (4) Based on this, a random angle perturbation is introduced: (5) In the formula, Indicates at time A random angle; Based on the unit direction away from the center of the encirclement and random angle perturbation The final escape direction of the escape target is: (6)。 3. The multi-unmanned surface vessel cooperative encirclement method integrating graph neural networks and attention as described in claim 2, characterized in that, In S2, the specific steps for obtaining the global state based on the multi-unmanned surface vessel (USV) collaborative encirclement mission scenario and the current pursuit status, obtaining the observation information of each pursuit USV based on the global state, and calculating the encirclement characteristics of the pursuit USV based on the observation information include: The coordinated encirclement task is modeled as a Markov decision process, and a state space is defined. This represents the global state at each moment; Set the observation model as The global state is mapped to local observations of each unmanned surface vessel (USV) using an observation model, thereby outputting the observation set of each pursuit USV. , This indicates the observation information of each pursuit drone; Based on the observation information of the pursuit unmanned surface vessel, the basic and extended features are calculated, including: The fundamental features include distance balance. and relative motion trend Among them, distance uniformity is used to measure the consistency of the distance distribution from each pursuing unmanned surface vessel to the escaping target, and the calculation formula is: (7) In the formula, Indicates the first The distance between the pursuing unmanned vessel and the escaping target. Indicates the number of unmanned vessels being pursued; The relative motion trend is used to characterize the degree of consistency between the current direction of motion of the escaping target and the direction of the encirclement center. The calculation formula is as follows: (8) In the formula, Indicates the geometric center of the pursuit unmanned surface vessel formation. , Indicates the first i The location of the pursuing unmanned vessel Indicates the direction of movement of the escaping target. , Indicates the heading angle of the escape target. Indicates the location of the escape target; The extended features include closure, maximum capture gap, capture quality, symmetry, distribution balance, and coverage integrity, among which, the first... k The formula for calculating the closure degree of a single pursuit unmanned surface vessel is: (9) In the formula, Indicates the first j The location of the pursuing unmanned vessel To pursue the number of unmanned boats; The maximum gap in the encirclement is used to extract the weakest link in the coordination during the encirclement. k The formula for calculating the maximum gap in the encirclement of a single pursuit unmanned vessel is: (10) The encirclement quality is used to characterize the overall encirclement quality when the escaping target deviates from the encirclement area. k The formula for calculating the capture mass of a single pursuit unmanned surface vessel is: (11) The symmetry is used to measure the relative centrality of the pursuit unmanned surface vessel within the formation. k The formula for calculating the symmetry of a single pursuit unmanned surface vessel is: (12) in, Let represent the set of numbered pairs consisting of all the other pursuit unmanned vessels except for the k-th one. Represents a set The number of numbered pairs Indicates the position of the k-th pursuit drone; The distribution uniformity is used to measure the uniformity of the angular distribution of each pursuit drone around the escaping target. k The formula for calculating the distribution balance of the pursuit unmanned surface vessels is: (13) In the formula, This indicates the angular interval between each pursuing unmanned vessel and the escaping target; The coverage integrity is used to characterize the largest uncovered corner area of the encirclement formation. k The formula for calculating the coverage integrity of a single pursuit drone is: (14) Therefore, the first The characteristics of the pursuit of unmanned surface vessels are represented as follows: (15) in, This represents the set of basic motion state features of the k-th pursuit unmanned surface vessel. Let K represent the set of extended encirclement geometric features of the k-th pursuit unmanned vessel. Including distance balance and relative motion trend , Including closure The largest gap in the encirclement Encirclement quality Symmetry Distribution balance and coverage completeness : (16)。 4. The multi-unmanned surface vessel cooperative encirclement method integrating graph neural networks and attention as described in claim 3, characterized in that, In S3, the specific steps for establishing a graph neural network based on the encirclement features of the pursuit unmanned surface vessel, performing feature weighting and aggregation based on the graph neural network, and outputting the graph aggregated features include: Building a graph neural network: At each decision-making moment, the pursuing unmanned surface vessel and the escaping target are abstracted as nodes in a graph neural network, and a directed graph is constructed, represented as: (17) In the formula, Indicates the escape target node; Represents a set of nodes. Represents the set of edges; The relative relationships between the nodes pursuing the unmanned surface vessel and the escape target node are used as graph edges, i.e., the adjacency matrix is defined as follows: (18) In the formula, This is the adjacency distance threshold; Based on the graph neural network, feature weighting aggregation is performed, and the output graph aggregated features include: By using a pre-built multilayer perceptron, the features of the pursuing unmanned surface vessel node and the escape target node in the graph neural network are nonlinearly encoded to obtain the encoded node hidden representations: (19) In the formula, Represents the ReLU activation function; and These represent the weight matrix and bias of the first fully connected layer in a multilayer perceptron, respectively. Represents a node v The original features have a dimension of 8; and These represent the weight matrix and bias of the second fully connected layer in a multilayer perceptron, respectively. Represents a 64-dimensional vector space; By using a pre-defined two-layer graph attention layer to perform weighted aggregation of adjacent node information, the updated node hidden representation is obtained. ; The updated node representation is hidden through a pre-defined output layer. The compressed graph aggregation features, represented as: (20) In the formula, and These represent the weight matrix and bias of the first fully connected layer in the output layer, respectively. and These represent the weight matrix and bias of the second fully connected layer in the output layer, respectively. This represents a 32-dimensional vector space.
5. The multi-unmanned surface vessel cooperative encirclement method according to claim 4, which integrates graph neural networks and attention, is characterized in that... In S4, the control actions of the pursuit unmanned surface vessel output by the policy network are represented as follows: (21) In the formula, Indicates speed control quantity. Indicates the angular velocity control quantity; The specific steps for estimating the output state value of the value network include: The query vector, key vector, and value vector are calculated and represented as follows: (22) In the formula, Input features, i.e., graph aggregation features ; These represent the weight matrices for the query vector, key vector, and value vector, respectively. The attention weights are calculated as follows: (23) In the formula, The feature dimensions represent the query vector and key vector; Represents the transpose of the key vector; Based on the above attention weights, the value vector V Perform weighted aggregation to obtain the output of each head; The outputs of each head are concatenated and linearly transformed to obtain the attention interaction features of all the pursuit unmanned surface vessels. The attention interaction features are subjected to residual connection and layer normalization to obtain the first fusion feature; The first fused feature is fed into a feedforward network consisting of two fully connected layers, ReLU activation and Dropout to perform nonlinear transformation and obtain nonlinear transformation features. The nonlinear transformation feature and the first fusion feature are subjected to residual connection and layer normalization to obtain the second fusion feature; The second fused feature is flattened and concatenated along the feature dimension, and then input into the value output network consisting of multiple fully connected layers and Tanh activation function to finally obtain the state value estimate.
6. The multi-unmanned surface vessel cooperative encirclement method according to claim 5, which integrates graph neural networks and attention, is characterized in that... In S5, the reward value is calculated based on the encirclement situation reflected by the new global state, including: At each moment, the reward value for each pursuit drone is calculated based on a set reward function, which is expressed as: (24) Among them, the reward for encirclement and capture Balanced rewards based on distance Evenly distributed angles reward Complete capture reward Step length penalty and collision penalty Weighted composition: (25) (26) (27) (28) In the formula, These are the weighting coefficients; This represents the average distance from all pursuing unmanned vessels to the escaping target. This represents the standard deviation of the corresponding distance. Indicates the radius of the encirclement. This represents the angular interval between adjacent pursuit unmanned surface vessels after sorting by azimuth; step size penalty. Fixed negative reward, collision penalty Take the penalty value when a collision occurs; Successful capture reward Represented as: (29) In the formula, This indicates the number of consecutive steps that meet the capture conditions. This indicates the number of consecutive steps required to determine if the encirclement is successful. Indicates the maximum number of steps. Indicates the current moment.