Cooperative hunting control method for unmanned surface vehicle

Abstract

The embodiment of the invention provides an unmanned ship cooperative hunting control method, and belongs to the technical field of control, and the method specifically comprises the steps: distributing a main hunting target for each chasing unmanned ship; for each chasing unmanned ship, an original observation vector containing a variable-length interaction sequence is formed according to the distributed main hunting target of the chasing unmanned ship; a variable-length interaction sequence in the original observation vector is converted into a standardized state vector of a fixed dimension through a two-stage attention mechanism; inputting the standardized state vector into a strategy network of the unmanned ship, and outputting a continuous action control instruction; and calculating an instant reward based on empirical data obtained after executing an action control instruction, and performing iterative updating on a strategy neural network and a corresponding value neural network by adopting a centralized training and decentralized execution mode based on a multi-agent depth deterministic strategy gradient framework to obtain a control model to generate a cooperative control scheme. Through the scheme disclosed by the invention, the control precision and adaptability are improved.

Description

Technical Field

[0001] This disclosure relates to the field of control technology, and in particular to a method for collaborative encirclement and capture control of unmanned surface vessels. Background Technology

[0002] Currently, with the expanding application of unmanned systems in the marine environment, unmanned surface vessels (USVs), as intelligent platforms with autonomous navigation and multi-task execution capabilities, are widely used in tasks such as marine exploration, environmental monitoring, resource inspection, maritime law enforcement, and search and rescue response. Multi-USV collaborative encirclement, as an important application scenario in USV swarm control, requires efficient coordination and precise decision-making capabilities among USVs. However, in complex marine environments and dynamically changing target scenarios, effectively achieving task allocation and dynamic formation encirclement remains a challenge for current research.

[0003] Existing task allocation strategies mainly include centralized optimization algorithms and distributed game theory mechanisms. While the former, such as genetic algorithms and particle swarm optimization, possesses certain global optimization capabilities, they typically have high computational complexity and insufficient support for real-time performance. The latter, although possessing parallelism and robustness, is prone to getting trapped in local optima, making it difficult to guarantee allocation quality and system resource utilization. Meanwhile, most traditional encirclement control methods are based on static formation structures, which are ill-suited to handling the complex movements of high-speed, intelligent targets and lack support for formation adaptability.

[0004] Existing research on unmanned surface vessel (USV) swarm control includes methods based on virtual force fields, model predictive control, and rule-driven mechanisms. While these methods have achieved some success in certain static or weakly dynamic environments, their success rate and efficiency in encirclement and capture against high-speed, intelligent escaping targets are insufficient to meet practical requirements. Especially in multi-target scenarios, the high degree of coupling between task allocation and encirclement coordination further increases the system's complexity.

[0005] In recent years, deep reinforcement learning has demonstrated powerful capabilities in agent control and game theory strategies, enabling distributed perception and decision-making control. However, handling variable-length observations and establishing effective information fusion mechanisms in multi-agent systems remains a technical challenge. Furthermore, the encirclement process places higher demands on formation flexibility and distribution uniformity, requiring more refined formation control strategies to address the dynamic escape behavior of targets.

[0006] It is evident that there is an urgent need for a collaborative capture and control method for unmanned surface vessels (USVs) that can enhance the mission adaptability and collaborative combat effectiveness of USV swarms in complex and dynamic tasks. Summary of the Invention

[0007] In view of this, the present disclosure provides a method for the cooperative capture and control of unmanned surface vessels, which at least partially solves the problems of poor control accuracy and adaptability in the prior art.

[0008] This disclosure provides a method for cooperative encirclement and control of unmanned surface vessels, including:

[0009] Step 1: For the multiple escape targets at the current moment, assign a main target to each pursuit drone and form multiple pursuit teams;

[0010] Step 2: For each pursuit unmanned vessel, based on its assigned main pursuit target, collect its own observation vector, the group situation observation vector with the main pursuit target, and the inter-vessel interaction observation vector with other pursuers in the cluster, forming an original observation vector containing a variable-length interaction sequence.

[0011] Step 3: Use a two-level attention mechanism to extract and aggregate features from the variable-length interaction sequence in the original observation vector, and convert it into a standardized state vector with a fixed dimension.

[0012] Step 4: Input the standardized state vector into the policy network of the unmanned surface vessel and output continuous motion control commands;

[0013] Step 5: Based on the experience data obtained after executing the action control command, calculate the immediate reward, which includes the target approach reward, collision avoidance reward, cooperative support reward, and adaptive formation reward. Accordingly, based on the multi-agent deep deterministic policy gradient framework, adopt a centralized training and decentralized execution approach to iteratively update the policy neural network and the corresponding value neural network to obtain the control model generating cooperative control scheme. The input of the value neural network includes the standardized state vectors of other pursuit unmanned surface vessels.

[0014] According to a specific implementation of an embodiment of this disclosure, step 1 specifically includes:

[0015] Step 1.1: Each pursuit drone calculates a comprehensive score based on its distance from each escape target and the number of pursuers near each escape target, and independently selects the target with the lowest score to complete the distributed initial allocation.

[0016] Step 1.2: After the initial allocation, identify high-density target groups with more than a preset threshold number of unmanned surface vessels (USVs) and low-density target groups with insufficient numbers.

[0017] Step 1.3: Establish an optimization model for redistributing the pursuit unmanned surface vessels from the high-density group to the low-density group, and use the particle swarm optimization algorithm to solve the model to achieve a balanced redistribution of task load.

[0018] According to a specific implementation of this disclosure, the self-observation vector includes the position, heading angle, and velocity of the vessel, as well as the relative position, relative velocity, heading angle difference, and distance deviation between the vessel and the main target.

[0019] The group situation observation vector includes the relative position with the global optimal pursuer and the deviation from the expected capture speed;

[0020] The relative position, relative speed, and heading difference between the inter-vessel interactive observation vector and each other pursuing vessel within a preset range.

[0021] According to a specific implementation of an embodiment of this disclosure, step 3 specifically includes:

[0022] Step 3.1: Use a linear layer to perform high-dimensional feature mapping on the variable-length interaction sequence;

[0023] Step 3.2: Input the mapped feature sequence into the Transformer encoder for processing;

[0024] Step 3.3: Input the output of the Transformer encoder into the weighted attention module and aggregate them to generate a fixed-length interactive feature vector;

[0025] Step 3.4: Concatenate the interactive feature vector with other fixed-length observation components in the original observation vector to form a standardized state vector.

[0026] According to one specific implementation of this disclosure, the continuous motion control command is a linear velocity control quantity and an angular velocity control quantity.

[0027] According to a specific implementation of an embodiment of this disclosure, the step of calculating the adaptive formation reward includes:

[0028] When the pursuit unmanned surface vessel is within the defined encirclement area centered on the escaping target, calculate the angle difference between it and the adjacent pursuit unmanned surface vessel;

[0029] Based on the angle difference and the number of pursuit drones entering the encirclement in the encirclement group, an adaptive factor for adjusting the orbiting speed is calculated.

[0030] Calculate the adaptive orbital velocity vector based on the maximum orbital velocity and the adaptive factor;

[0031] Calculate the adaptive formation bonus by combining the current velocity with the adaptive orbital velocity vector.

[0032] The unmanned surface vessel (USV) collaborative pursuit and control scheme in this embodiment includes: Step 1, assigning a primary pursuit target to each of the multiple escaping targets at the current moment, forming multiple pursuit teams; Step 2, for each USV, collecting its own observation vector, the group situation observation vector with the primary pursuit target, and the inter-vessel interaction observation vector with other pursuers in the cluster, based on its assigned primary pursuit target, to form an original observation vector containing a variable-length interaction sequence; Step 3, extracting and aggregating features from the variable-length interaction sequence in the original observation vector through a two-level attention mechanism, converting it into a fixed-dimensional standardized vector. Step 4: Input the standardized state vector into the policy network of the unmanned surface vessel (USV) and output continuous action control commands. Step 5: Based on the empirical data obtained after executing the action control commands, calculate the immediate reward, which includes the target approach reward, collision avoidance reward, cooperative support reward, and adaptive formation reward. Based on this, using a multi-agent deep deterministic policy gradient framework, adopt a centralized training and decentralized execution approach to iteratively update the policy neural network and the corresponding value neural network to obtain a cooperative control scheme for generating the control model. The input of the value neural network includes the standardized state vectors of other pursuit USVs.

[0033] The beneficial effects of the embodiments of this disclosure are as follows: the solution of this disclosure improves the efficiency and rationality of task allocation in a multi-objective environment through a two-stage task allocation mechanism, effectively improving the overall resource allocation performance of the system; after introducing a two-level attention structure, the network model has the ability to process variable-length observation inputs, significantly enhancing the generalization ability and dynamic environment adaptability of the learning strategy; the constructed dynamic formation reward mechanism can guide the pursuit unmanned surface vessel to form an encirclement structure with directional consistency and spatial uniformity, improving the stability of the encirclement strategy, and has good strategy convergence characteristics and adaptability to complex task scenarios, with strong engineering deployment feasibility and expansion potential. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating a collaborative encirclement and control method for unmanned surface vessels provided in this embodiment of the present disclosure;

[0036] Figure 2 This is a schematic diagram of a two-dimensional motion model of an unmanned surface vessel provided in an embodiment of the present disclosure;

[0037] Figure 3This is a comparative diagram of distributed and hybrid task allocation results provided in an embodiment of the present disclosure, wherein (a) represents distributed and (b) represents hybrid;

[0038] Figure 4 A strategy network structure diagram provided in this embodiment of the disclosure;

[0039] Figure 5 A value network structure diagram provided for embodiments of this disclosure;

[0040] Figure 6 A flowchart of a multi-agent deep reinforcement learning training process is provided in an embodiment of this disclosure;

[0041] Figure 7 This is a comparison chart of the reward curves of the embodiments of this disclosure and existing multi-agent algorithms during the training phase;

[0042] Figure 8 The diagrams provided in this disclosure illustrate the multi-target capture process under different escape strategies, where (a) represents the stationary mode, (b) represents the linear mode, (c) represents the random mode, and (d) represents the artificial potential field motion mode. Detailed Implementation

[0043] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0044] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0045] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0046] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0047] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0048] This disclosure provides a method for collaborative encirclement and control of unmanned surface vessels (USVs), which can be applied to the collaborative control of USVs in scenarios such as marine exploration, environmental monitoring, resource inspection, maritime law enforcement, and search and rescue response.

[0049] See Figure 1 This is a flowchart illustrating a method for cooperative encirclement and control of unmanned surface vessels provided in an embodiment of this disclosure. Figure 1 As shown, the method mainly includes the following steps:

[0050] Step 1: For the multiple escape targets at the current moment, assign a main target to each pursuit drone and form multiple pursuit teams;

[0051] Furthermore, step 1 specifically includes:

[0052] Step 1.1: Each pursuit drone calculates a comprehensive score based on its distance from each escape target and the number of pursuers near each escape target, and independently selects the target with the lowest score to complete the distributed initial allocation.

[0053] Step 1.2: After the initial allocation, identify high-density target groups with more than a preset threshold number of unmanned surface vessels (USVs) and low-density target groups with insufficient numbers.

[0054] Step 1.3: Establish an optimization model for redistributing the pursuit unmanned surface vessels from the high-density group to the low-density group, and use the particle swarm optimization algorithm to solve the model to achieve a balanced redistribution of task load.

[0055] In specific implementation, such as Figure 2 As shown, consider a two-dimensional inertial coordinate system The status of the unmanned surface vessel operating in the water is determined by its position. With heading angle This indicates that the control input is linear velocity. and angular velocity Its continuous-time motion model is as follows:

[0056]

[0057] in, and These represent the linear velocity and angular velocity of the unmanned surface vessel in the hull coordinate system, respectively.

[0058] To reflect the physical limitations of the actuator, the system control variables must meet the following constraints:

[0059]

[0060] in, and These are the maximum linear velocity and the maximum angular velocity, respectively. and These are the upper limits for linear acceleration and angular acceleration. The values ​​used in the experiment are as follows:

[0061]

[0062] This model provides a dynamic basis and control feasibility boundary for subsequent training of multi-agent cooperative strategies.

[0063] To address the problem of efficient matching between unmanned surface vessels (USVs) and escaping targets in multi-target encirclement scenarios, a hybrid task allocation method combining distributed initial selection and centralized optimization is proposed. The specific implementation steps are as follows.

[0064] (1) Fast distributed initial allocation

[0065] In the initial phase of the mission, each pursuit drone calculates a comprehensive score function based on its distance from each target and the number of existing drones near the target, as follows:

[0066]

[0067] in, unmanned surface vessel With the goal European distance, For the goal The number of pursuit drones within the perception range, and This is the weighting factor.

[0068] Each unmanned surface vessel independently selects the objective with the lowest score:

[0069]

[0070] This forms the initial allocation result. This phase requires no global information and can achieve efficient response, but may lead to uneven distribution, such as... Figure 3 As shown in (a).

[0071] (2) Centralized redistribution optimization

[0072] If, after initial allocation, there are too many or too few target groups for any given group, the system triggers a local centralized optimization mechanism. This mechanism reorganizes the pursuit teams based on whether the number of members exceeds a certain threshold. The data was divided into high-density and low-density groups. The optimized results are as follows: Figure 3 As shown in (b).

[0073] Assume the number of high-density groups is The number of low-density groups is Definition of the first The number of pursuers in each high-density group was The task redistribution can be modeled as the following optimization problem.

[0074] Let the distribution matrix of pursuers from high-density groups to low-density groups be:

[0075]

[0076] in, Indicates the first The first high-density group The pursuers were assigned to the... A low-density group, This represents the corresponding distance.

[0077] Define the decision vector and distance vector of the pursuer in the high-density group as follows:

[0078]

[0079] in, It is aimed at the first The optimized decision vector of a high-density population, if , indicating the tracker If a new goal is chosen, then ;otherwise, . This represents the distance matrix in the initial allocation scheme. This indicates the distance between the tracker and the selected target.

[0080] The final optimization objective function is:

[0081]

[0082] in, This represents the trace operation of a matrix. The constraints are as follows:

[0083] Each pursuer can be reassigned to a maximum of one new target:

[0084]

[0085] The number of pursuers after high-density group redistribution must not be lower than the average:

[0086]

[0087] Each low-density group should be replenished to exactly the average number of pursuers after redistribution:

[0088]

[0089] in, Indicates low-density group The initial number of pursuers assigned.

[0090] To solve the aforementioned combinatorial optimization problem, this invention employs the particle swarm optimization algorithm. This method, with its powerful global search capability, can effectively escape local optima traps, thereby obtaining a better global task allocation solution.

[0091] Step 2: For each pursuit unmanned vessel, based on its assigned main pursuit target, collect its own observation vector, the group situation observation vector with the main pursuit target, and the inter-vessel interaction observation vector with other pursuers in the cluster, forming an original observation vector containing a variable-length interaction sequence.

[0092] Furthermore, the self-observation vector includes the position, heading angle, and speed of the vessel, as well as the relative position, relative speed, heading angle difference, and distance deviation between the vessel and the main target.

[0093] The group situation observation vector includes the relative position with the global optimal pursuer and the deviation from the expected capture speed;

[0094] The relative position, relative speed, and heading difference between the inter-vessel interactive observation vector and each other pursuing vessel within a preset range.

[0095] In specific implementation, this invention designs the observation space and action space of the unmanned surface vessel based on the Multi-Agent Deep Reinforcement Learning (MADRL) framework to support its strategy learning and collaborative decision-making in multi-target dynamic encirclement scenarios.

[0096] The observation space consists of the following three types of information:

[0097] Self-observation information: including the relative position, relative speed, heading angle difference, and target distance deviation between the pursuing unmanned surface vessel and the target, denoted as... ;

[0098] Group situational information: describes the relative position of the current individual to the globally optimal pursuer and the expected encirclement speed. The deviation is used to guide the consistency of collective encirclement and capture, and is expressed as ;

[0099] Inter-vessel information: Depicting the relative position, speed, heading difference, and safe distance deviation between the vessel and surrounding pursuers, expressed as... .

[0100] The final observation vector is constructed as follows:

[0101]

[0102] The motion space employs continuous control, with the controlled variables being velocity and angular velocity. Its conversion into system control commands is as follows:

[0103]

[0104] in, The deviation angle between the current heading and the target heading. To control the cycle.

[0105] Step 3: Use a two-level attention mechanism to extract and aggregate features from the variable-length interaction sequence in the original observation vector, and convert it into a standardized state vector with a fixed dimension.

[0106] Based on the above embodiments, step 3 specifically includes:

[0107] Step 3.1: Use a linear layer to perform high-dimensional feature mapping on the variable-length interaction sequence;

[0108] Step 3.2: Input the mapped feature sequence into the Transformer encoder for processing;

[0109] Step 3.3: Input the output of the Transformer encoder into the weighted attention module and aggregate them to generate a fixed-length interactive feature vector;

[0110] Step 3.4: Concatenate the interactive feature vector with other fixed-length observation components in the original observation vector to form a standardized state vector.

[0111] In practical implementation, to address the problem of inconsistent observation information length caused by the dynamic change in the number of the pursuer's neighbors, this invention proposes a strategy and evaluation network design scheme with a two-level attention fusion structure.

[0112] The structure of a policy network is as follows Figure 4 As shown, the inter-submarine interactive observation sequence First, high-dimensional feature mapping is performed through a linear layer. Then, the sequence sequentially enters a dual-attention module consisting of a Transformer encoder and a weighted attention mechanism to extract key topological relationships and behavioral features.

[0113] Value network structure such as Figure 5 As shown, its overall structure is similar to the policy network, but it is used to process joint state-action pairs for all agents, significantly increasing the dimensionality of the input information, which is represented as a variable-length sequence. To accurately model the complex dependencies between individuals, this network expands the linear transformation layer and the size of the hidden nodes to improve the fitting accuracy of the action-value function and significantly accelerate the convergence speed during training.

[0114] Step 4: Input the standardized state vector into the policy network of the unmanned surface vessel and output continuous motion control commands;

[0115] Furthermore, the continuous motion control command is a linear velocity control quantity and an angular velocity control quantity.

[0116] In practice, after feature aggregation, the output is mapped to a fixed-length vector and compared with the self-observation vector. and group-level observation vectors The data is then pieced together to form a complete state input. Finally, this state information is used by a multilayer perceptron (MLP) to generate continuous action outputs, ensuring the accuracy and stability of the control commands.

[0117] Step 5: Based on the experience data obtained after executing the action control command, calculate the immediate reward, which includes the target approach reward, collision avoidance reward, cooperative support reward, and adaptive formation reward. Accordingly, based on the multi-agent deep deterministic policy gradient framework, adopt a centralized training and decentralized execution approach to iteratively update the policy neural network and the corresponding value neural network to obtain the control model generating cooperative control scheme. The input of the value neural network includes the standardized state vectors of other pursuit unmanned surface vessels.

[0118] Based on the above embodiments, the step of calculating the adaptive formation reward includes:

[0119] When the pursuit unmanned surface vessel is within the defined encirclement area centered on the escaping target, calculate the angle difference between it and the adjacent pursuit unmanned surface vessel;

[0120] Based on the angle difference and the number of pursuit drones entering the encirclement in the encirclement group, an adaptive factor for adjusting the orbiting speed is calculated.

[0121] Calculate the adaptive orbital velocity vector based on the maximum orbital velocity and the adaptive factor;

[0122] Calculate the adaptive formation bonus by combining the current velocity with the adaptive orbital velocity vector.

[0123] In practical implementation, to achieve rapid, stable, and collaborative target acquisition, this invention designs multiple composite reward functions to guide the policy network to learn efficiently. Specifically, it includes the following four parts:

[0124] Approaching Target Reward: This reward function is designed to encourage pursuers to approach the target as quickly as possible and effectively form an encirclement. The reward for approaching the target is adjusted based on the distance between the pursuer and the target, and is defined as follows:

[0125]

[0126] in, For the desired capture radius, This represents the actual distance between the pursuer and the target.

[0127] Collision Penalty Reward: To ensure a safe distance between the pursuer and the target, as well as other pursuers, this reward function penalizes approaching or colliding actions to ensure system safety. The specific expression is:

[0128]

[0129] in, and These represent the collision distance error between the pursuer and the target, and the collision distance error between the pursuer and other pursuers, respectively. It is a step function.

[0130] Collaborative Support Rewards: To promote cooperation among pursuers, group support rewards are designed. When pursuers are far from the target, they can rely on guidance from other pursuers to quickly approach the target, ensuring the efficiency of the encirclement. The specific function is as follows:

[0131]

[0132] in, To support distance thresholds, This represents the distance between the pursuer and the optimal pursuer.

[0133] Adaptive Formation Reward: When the pursuer is in the encirclement area Within this timeframe, the system guides the target to achieve balanced angular distribution and rotational encirclement behavior. The encirclement reward is defined as:

[0134]

[0135] in, and The inner and outer radii of the enclosure area, At the current speed, The adaptive orbital velocity vector is defined as follows:

[0136]

[0137] in, For maximum orbital speed, The adaptive factor, which reflects the uniformity of angular distribution, is defined as:

[0138]

[0139] in These represent the angular differences between the current pursuer and its forward and backward adjacent vessels, respectively. This represents the number of pursuers currently in the encirclement group. This mechanism achieves self-adjusting behavior by accelerating in sparse areas and decelerating in dense areas.

[0140] , , , , The adjustment coefficients for each reward are used, and the final instant reward is the weighted sum of all sub-rewards:

[0141]

[0142] in, Assign weights to each reward, and .

[0143] Overall training framework as follows Figure 6As shown, a centralized training-decentralized execution (CTDE) strategy is employed to achieve efficient learning and stable convergence in a multi-target pursuit environment. At each time step, the system first dynamically assigns the pursuer to the corresponding target using the hybrid task allocation module. Subsequently, each pursuer outputs control actions through its policy network, and exploratory noise is added to enhance the policy's exploratory nature. The generated joint actions interact with the environment to obtain state transition tuples. The data is then stored in the experience replay buffer. Subsequently, the adaptive reward module evaluates the performance scores of each pursuer based on the real-time status of the encirclement task. During training, mini-batch samples are randomly sampled from the experience pool and used to update both the policy network and the value network. Furthermore, the target network employs a soft update mechanism to enhance the robustness of the training process. This training mechanism enables effective collaboration among pursuers and possesses strong policy transfer capabilities and adaptability to dynamic target behavior.

[0144] The unmanned surface vessel (USV) collaborative encirclement and control method provided in this embodiment improves the efficiency and rationality of task allocation in multi-objective environments through a two-stage task allocation mechanism, effectively enhancing the overall resource allocation performance of the system. After introducing a two-level attention structure, the network model has the ability to handle variable-length observation inputs, significantly enhancing the generalization ability of the learning strategy and its adaptability to dynamic environments. The constructed dynamic formation reward mechanism can guide the pursuing USV to form an encirclement structure with directional consistency and spatial uniformity, improving the stability of the encirclement strategy. It also has good strategy convergence characteristics and adaptability to complex task scenarios, and has strong engineering deployment feasibility and expansion potential.

[0145] The method of this disclosure will be further described below with reference to a specific embodiment. The technical solution provided by this invention includes the following steps:

[0146] (1) Multi-target task allocation: A hybrid allocation mechanism is proposed. In the first stage, efficient initial matching is achieved through a distributed strategy. In the second stage, a centralized optimization algorithm is introduced to correct the initial allocation results to ensure that each target is allocated a sufficient number of pursuers, thereby achieving resource load balancing and allocation stability.

[0147] (2) Construction of observation space: Design a hierarchical observation structure, including self-observation (such as relative distance, relative speed and heading difference with the target), inter-vessel observation (such as relative pose and distance tolerance) and cluster situation observation (such as relative state and expected speed difference with the best individual), to support high-dimensional feature fusion and strategy determination.

[0148] (3) Action space modeling: By mapping the policy output to the control increments of velocity direction and velocity amplitude, the nonlinear coupling relationship between state and action is reduced, thereby improving the controllability and convergence stability of policy training.

[0149] (4) Fusion of dual-level attention mechanism: To address the problem of dynamic changes in the length of neighbor observation sequences, the Transformer encoder and weighted attention module are fused to extract and aggregate the features of variable-length sequences, and finally form a fixed-dimensional state representation, which effectively improves the input robustness and decision generalization ability of the model.

[0150] (5) Reward function design: Construct a multi-component composite reward function system, taking into account target proximity, safety distance constraints, collaborative tracking behavior and dynamic formation consistency. Through reward guidance, the convergence direction and inter-individual cooperation strategies in the reinforcement learning process are strengthened, so as to promote the stable operation of the system.

[0151] (6) Policy training and control implementation: Under the MADDPG framework, the policy network and evaluation network with dual attention structure are trained respectively. The centralized training and decentralized execution mechanism is adopted, and the experience replay and soft update strategy are combined to improve the convergence speed of the model and the policy transfer ability in complex scenarios.

[0152] To verify the effectiveness and generalization ability of the multi-target pursuit-escape control strategy proposed in this invention, a two-dimensional simulation environment based on the OpenAI Gym platform was constructed, with the simulation region having a side length of [missing information]. A square body of water, with no obstacles within the area. The initial positions of all pursuers and targets are randomly generated, and the maximum speeds of the pursuers and escapees are set to [values ​​to be filled in]. and All experiments were conducted on a computing platform configured with an Intel i7-12700F CPU and an NVIDIA RTX 4060Ti GPU, using PyTorch as the reinforcement learning framework.

[0153] (1) Comparison of training process and strategy

[0154] During the training phase, the scenario involves four agents pursuing a single target, which dynamically escapes using an Artificial Potential Field (APF) strategy. Each agent learns a policy based on perceptual information and outputs action commands through a policy network. During training, the proposed DAT-MADDPG method is compared with existing multi-agent reinforcement learning algorithms. Figure 7 As shown, the method of the present invention exhibits rapid growth in returns in the early stages of training and eventually converges to a higher policy reward level, demonstrating excellent policy learning efficiency and convergence performance.

[0155] (2) Verification of multi-target encirclement capability

[0156] To test the algorithm's multi-target adaptability, a complex adversarial scenario involving multiple targets and multiple pursuers was constructed. The initial task was handled by a hybrid task allocation mechanism to achieve dynamic load balancing. Figure 8 The experiment demonstrates the pursuit process of a task force in a multi-target scenario. The experiment covers various target escape modes, including stationary, linear, random, and APF, and verifies the stability and universality of the method in this invention under multi-task conditions.

[0157] In summary, the multi-agent deep reinforcement learning method based on a two-level attention mechanism proposed in this invention demonstrates excellent encirclement performance under various target escape strategies. It exhibits rapid convergence during the training phase and strong generalization and scalability during the testing phase, making it suitable for practical tasks such as multi-target cooperative encirclement in complex marine environments.

[0158] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0159] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for multi-unmanned surface vehicle cooperative hunting control, characterized in that, The method comprises the following steps: Step 1, for a plurality of escape targets at the current moment, assigning each of the pursuing unmanned ships a main surrounding target to form a plurality of pursuit teams; Step 2, for each of the pursuing unmanned ships, collecting its own observation vector, the group situation observation vector of the main surrounding target and the inter-ship interaction observation vector of other pursuers in the cluster to form an original observation vector containing a variable-length interaction sequence; Step 3, extracting and aggregating the variable-length interaction sequence in the original observation vector through a double-stage attention mechanism to convert it into a standardized state vector of a fixed dimension; Step 4, inputting the standardized state vector into the strategy network of the unmanned ship to output continuous action control instructions; Step 5, based on the experience data obtained after executing the action control instructions, calculating an immediate reward including an approach target reward, a collision avoidance reward, a cooperative support reward and an adaptive formation reward, and based on the multi-agent deep deterministic policy gradient framework, iteratively updating the strategy neural network and the corresponding value neural network in a centralized training and decentralized execution manner to obtain a control model to generate a cooperative control scheme, wherein the input of the value neural network includes the standardized state vectors of other pursuing unmanned ships.

2. The method of claim 1, wherein, The step 1 specifically comprises: Step 1.1, making each of the pursuing unmanned ships calculate a comprehensive score according to the distances of the ships from each of the escape targets and the number of pursuers near each of the escape targets, and independently selecting the target with the smallest score to complete the distributed initial assignment; Step 1.2, after the initial assignment, identifying a high-density target group with a number of pursuing unmanned ships exceeding a preset threshold and a low-density target group with an insufficient number of pursuing unmanned ships; Step 1.3, establishing an optimization model for reassigning pursuing unmanned ships from the high-density group to the low-density group, and solving the model by using a particle swarm optimization algorithm to achieve task load balancing reassignment.

3. The method of claim 2, wherein, The own observation vector includes the position, heading angle and speed of the ship, and the relative position, relative speed, heading angle difference and distance deviation of the ship from the main surrounding target; The group situation observation vector includes the relative position of the global optimal pursuer and the deviation from the expected surrounding speed; The inter-ship interaction observation vector includes the relative position, relative speed and heading difference of each of the other pursuing ships within a preset range.

4. The method of claim 3, wherein, The step 3 specifically comprises: Step 3.1, using a linear layer to perform high-dimensional feature mapping on the variable-length interaction sequence; Step 3.2, inputting the mapped feature sequence into a Transformer encoder for processing; Step 3.3, inputting the output of the Transformer encoder into a weighted attention module to aggregate a fixed-length interaction feature vector; Step 3.4, concatenating the interaction feature vector with other fixed-length observation components in the original observation vector to form a standardized state vector.

5. The method of claim 1, wherein, The continuous action control instruction is a linear velocity control quantity and an angular velocity control quantity.

6. The method of claim 1, wherein, The step of calculating the adaptive formation reward comprises: When the pursuing unmanned ship is in a defined surrounding area centered on the escape target, calculating the angle difference between the ship and the adjacent pursuing unmanned ship; According to the angle difference and the number of the pursuit unmanned ships entering the trapping circle in the trapping group, an adaptive factor for adjusting the surrounding speed is calculated; According to the maximum surrounding speed and the adaptive factor, an adaptive surrounding speed vector is calculated; Combined with the current speed and the adaptive surrounding speed vector, an adaptive formation reward is calculated.

Images